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Abstract 

We consider a dynamic server allocation problem over parallel queues with randomly varying connectivity and server 
switchover delay between the queues. At each time slot the server decides either to stay with the current queue or switch to another 
queue based on the current connectivity and the queue length information. Switchover delay occurs in many telecommunications 
applications and is a new modeling component of this problem that has not been previously addressed. We show that the 
simultaneous presence of randomly varying connectivity and switchover delay changes the system stability region and the structure 
of optimal policies. In the first part of the paper, we consider a system of two parallel queues, and develop a novel approach to 
explicitly characterize the stability region of the system using state-action frequencies which are stationary solutions to a Markov 
?H ' Decision Process (MDP) formulation. We then develop a frame-based dynamic control (FBDC) policy, based on the state-action 

^ ' frequencies, and show that it is throughput-optimal asymptotically in the frame length. The FBDC policy is applicable to a broad 

class of network control systems and provides a new framework for developing throughput-optimal network control policies using 
state-action frequencies. Furthermore, we develop simple Myopic policies that provably achieve more than 90% of the stability 
region. 

In the second part of the paper we extend our results to systems with an arbitrary number of queues. In particular, we show 
^ that the stability region characterization in terms of state-action frequencies and the throughput-optimality of the FBDC policy 

V,^ . follow for the general case. Furthermore, we characterize an outer bound on the stability region and an upper bound on sum- 

C^ ■ throughput and show that a simple Myopic policy can achieve this sum-throughput upper-bound in the corresponding saturated 

^^ ' system. Finally, simulation results show that the Myopic policies may achieve the full stability region and are more delay efficient 

r^ ' than the FBDC policy in most cases. 

-(— > I 
C^ \ Index Terms 

Switchover delay, randomly varying connectivity, scheduling, queueing, switching delay, Markov Decision Process, uplink, 
downlink, wireless networks 
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I. Introduction 



^«0 ' Scheduling a dynamiql server over time varying wireless channels is an important and well-studied research problem which 

^ ! provides useful mathematical modeling for many practical applications |Il,|[l7l,|l2l,||29l,|l32l,|l33l, El-liOl, ll45l. B6ll. 

^0—^ ' However, to the best of our knowledge, the joint effects of randomly varying connectivity and server switchover delay have 
• , not been considered previously. In fact, switchover delay is a widespread phenomenon that can be observed in many practical 

ilJ ' network systems. In satellite systems where a mechanically steered antenna is providing service to ground stations, the time 

*vj to switch from one station to another can be around 10ms ||9], BTI . Similarly, the delay for electronic beamforming can be 
more than 300/is in wireless radio systems ||3], ||9], PTI . and in optical communication systems tuning delay for transceivers 
can take significant time (/is-ms) ITTJI . ||28l . 

We consider the dynamic server control problem for parallel queues with time varying channels and server switchover delay 

^^ ,, as shown in Fig. [T] We consider a slotted system where the slot length is equal to a single packet transmission time and it 
H ■ takes one slot for the server to switch from one queue to anotheo One packet is successfully received from queue i if the 

- - - server is currently at queue i, it decides to stay at queue i, and queue i is connected, where i G {1,2, ..., A^} and N is the 
total number of queues. The server dynamically decides to stay with the current queue or switch to another queue based on 
the connectivity and the queue length information. Our goal is to study the impact of the simultaneous presence of switchover 
delays and randomly varying connectivity on system stability and to develop optimal control algorithms. We show that the 
stability region changes as a function of the memory in the channel processes, and it is significantly reduced as compared to 
systems without switchover delay. Furthermore, we show that throughput-optimal pohcies take a very different structure from 
the celebrated Max- Weight algorithm or its variants. 

A. Main Results 

In the first part of the paper we consider a two-queue system and develop fundamental insights for the problem. We first 
consider the case of memoryless (i.i.d.) channels where we characterize the stability region explicitly and show that simple 
Exhaustive type policies that ignore the current queue size and channel state information are throughput-optimal. Next we 
consider the Gilbert-Elliot channel model |[T], ||20l which is a commonly used model to abstract physical channels with 
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'in a slotted system, even a minimal switchover delay will lead to a loss of a slot due to synchronization issues. 
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Fig. 1: System model. Pai'allel queues with randomly varying connectivity processes Ci(t),C2(i), ...,Cjv(i) and ts = 1 slot switchover time. 



memory. We develop a novel methodology to characterize the stability region of the system using state-action frequencies 
which are steady-state solutions to an MDP formulation for the corresponding saturated system, and characterize the stability 
region explicitly in terms of the connectivity parameters. Using this state-action frequency approach, we develop a novel 
frame-based dynamic control (FBDC) policy and show that it is throughput-optimal asymptotically in the frame length. Our 
FBDC policy is the only known policy to stabilize systems with randomly varying connectivity and switchover delay and it 
is novel in that it utilizes the state-action frequencies of the MDP formulation in a dynamic queuing system. Moreover, we 
develop a simple 1-Lookahead Myopic policy that provably achieves at least 90% of the stability region, and myopic policies 
with 2 and 3 lookahead that achieve more than 94% and 96% of the stability region respectively. Finally, we present simulation 
results suggesting that the myopic policies may be throughput-optimal and more delay efficient than the FBDC policy. 

In the second part of the paper we consider the general model with arbitrary number of parallel queues. For memoryless 
(i.i.d.) channel processes we explicitly characterize the stability region and the throughput-optimal policy. For channels with 
memory, we show that the stability region characterization in terms of state-action frequencies extend to the general case and 
establish a tight outer bound on the stability region and an upper bound on the sum-throughput explicitly in terms of the 
connectivity parameters. We quantify the switching loss in sum-throughput as compared to the system with no switchover 
delays and show that simple myopic policies achieve the sum-throughput upper bound in the corresponding saturated system. 
We also show that the throughput-optimality of the FBDC policy extend to the general case. In fact, the FBDC policy provides a 
new framework for achieving throughput-optimal network control by applying the state-action frequencies of the corresponding 
saturated system over frames in the dynamic queueing system. The FBDC policy is applicable to a broad class of systems 
whose corresponding saturated model is Markovian with a weakly communicating and finite state space, for example, systems 
with arbitrary switchover delays (i.e., systems that take any finite number of time slots for switching) and general Markov 
modulated channel processes. Moreover, the framework of the FBDC policy can be utilized to achieve throughput-optimality 
in systems without switchover delay, for instance, in classical network control problems such as those considered in ll33l , 1361 , 

B. Related Work 

Optimal control of queuing systems and communication networks has been a very active research topic over the past two 
decades HT), ES], ||29l, ED, IIS, EH-lgOl, lH, EH. In the seminal paper ||39l, Tassiulas and Ephremides characterized 
the stability region of multihop wireless networks and proposed the throughput-optimal Max- Weight scheduling algorithm. 
In Pol , the same authors considered a parallel queuing system with randomly varying connectivity where they characterized 
the stability region of the system explicitly and proved the throughput-optimality of the Longest-Connected-Queue scheduling 
policy. These results were later extended to joint power allocation and routing in wireless networks in ll32l . Il33l and optimal 
scheduling for switches in 1361 . Il38l . More recently, suboptimal distributed scheduling algorithms with throughput guarantees 
were studied in lT3l . Il22l . l25l . l45l . while ifTTl . l29l developed distributed algorithms that achieve throughput-optimality (see 
lT9l . l30l for a detailed review). The effect of delayed channel state information was considered in IISTI . ||37| . ||46l which 
showed that the stability region is reduced and that a policy similar to the Max- Weight algorithm is throughput-optimal. 

Perhaps the closest problem to ours is that of dynamic server allocation over parallel channels with randomly varying 
connectivity and limited channel sensing that has been investigated in ||T|, ll24l . [|47l under the Gilbert-Elliot channel model. 
The saturated system was considered and the optimality of a myopic policy was established for a single server and two 
channels in ||47]| , for arbitrary number of channels in [IT], and for arbitrary number of channels and servers in (0. The problem 
of maximizing the throughput in the network while meeting average delay constraints for a small subset of users was considered 
in l3Tl . The average delay constraints were turned into penalty functions in 13T1 and the the theory of Stochastic Shortest Path 
problems, which is used for solving Dynamic Programs with certain special structures, was utilized to minimize the resulting 



drift+penalty terms. Finally, a partially observable Markov decision process (POMDP) model was used in |16;| to analyze 
dynamic multichannel access in cognitive radio systems. However, none of these existing works consider the server switchover 
delays. 

Switchover delay has been considered in Polling models in queuing theory community (e.g., Q, ED, ll26l . B^ t. however, 
randomly varying connectivity was not considered since it may not arise in classical Polling applications. Similarly, scheduling 
in optical networks under reconfiguration delay was considered in ifTTl . ifTSl . again in the absence of randomly varying 
connectivity, where the transmitters and receivers were assumed to be unavailable during reconfiguration. A detailed survey of 
the works in this field can be found in 142^. To the best of our knowledge, this paper is the first to simultaneously consider 
random connectivity and server switchover times. 

C. Main Contribution and Organization 

The main contribution of this paper is solving the scheduling problem in parallel queues with randomly varying connectivity 
and server switchover delays for the first time. For this, the paper provides a novel framework for solving network control 
problems via characterizing the stability region in terms of state-action frequencies and achieving throughput-optimality by 
utilizing the state-action frequencies over frames. 

This paper is organized as follows. We consider the two-queue system in Section |II] where we characterize the stability 
region together with the throughput-optimal policy for memoryless channels. We develop the state-action frequency framework 
in Section III-CI for channels with memory and use it to explicitly characterize the system stability region. We prove the 
throughput-optimality of the FBDC policy in Section III-DI and analyze simple myopic policies in Section III-EI We extend our 
results to the general case in Section |III] where we also develop outer bounds on the stability region and an upper bound on 
the sum-throughput achieved by a simple Myopic policy. We present simulation results in Section |IV] and conclude the paper 
in Section W\ 

II. Two-Queue System 
A. System Model 

Consider two parallel queues with time varying channels and one server receiving data packets from the queues. Time is 
slotted into unit-length time slots equal to one packet transmission time; t e {0,1,2,...}. It takes one slot for the server 
to switch from one queue to the other, and m{t) denotes the queue at which the server is present at slot t. Let the i.i.d. 
stochastic process Ai (t) with average arrival rate Xi denote the number of packets arriving to queue i at time slot t, where 
^[Af{t)] < A^ax' * S {1, 2}. Let C(t) == (Ci(i), C2(i)) be the channel (connectivity) process at time slot t, where C,(i) = 
for the OFF state (disconnected) and Ci{t) = 1 for the ON state (connected). We assume that the processes Ai{t), A2{t), Ci{t) 
and C2{t) are independent. 

The process Ci{t),i e {1,2}, is assumed to form the two-state Markov chain with transition probabilities pio and poi as 
shown in Fig. E i.e., the Gilbert-Elliot channel model Q, 1201, Ell, 113, El- The Gilbert-Elliot Channel model has been 
commonly used in modeling and analysis of wireless channels with memory JT], ll24l . ll44l . BtI . Bsl . For ease of exposition, 
we present the analysis in this section for the symmetric Gilbert-Elliot channel model, i.e., pio = Poi = e, and we state the 
corresponding results for the non-symmetric case in Appendix B. The steady state probability of each channel state is equal to 
0.5 in the symmetric Gilbert-ElUot channel model. Moreover, for e = 0.5, Ci{t) — l,w.p. 0.5, independently and identically 
distributed (i.i.d.) at each time slot. We refer to this case as the memoryless channels case. 

Let Q(t) = ((5i(t), (52(0) be the queue lengths at time slot t. We assume that Q(<) and C{t) are known to the server at 
the beginning of each time slot. Let at G {0, 1} denote the action taken at the beginning of slot t, where at = 1 if the server 
stays with the current queue and at = if it switches to the other queue. One packet is successfully received from queue i at 
time slot t, if m(t) = z, at = 1 and Ci{t) ~ 1. 

Definition 1 (Stability h30\l , 432 1/ )." The system is stable if 

1 *~^ 
limsup-^ ^ E[Qj(t)] < 00. 

•r=0jG{l,2} 

For the case of integer valued arrival processes, this stability criterion implies the existence of a long run stationary distribution 
for the queue sizes with bounded first moments ||30| . 

Definition 2 (Stability Region l[30]l , 452 1/ )." The stability region A is the set of all arrival rate vectors X — (Ai, A2) such that 
there exists a control algorithm that stabilizes the system. 

The (5-stripped stability region is defined for some (5 > as A* = -^ (Ai, A2)|(Ai +5, \2 + 5} G A >. A policy is said to achieve 
7-fraction of A, if it stabilizes the system for all input rates inside 7A, where 7=1 for a throughput-optimal policy. 

In the following, we start by explicitly characterizing the stability region for both memoryless channels and channels with 
memory and show that channel memory can be exploited to enlarge the stability region significantly. 
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Fig. 2: Markov modulated ON/OFF channel process. We have pio +poi < 1 (e < 0.5) for positive correlation. 
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Fig. 3: Stability region under memoryless (i.i.d.) channels and channels with memory (Markovian with e < 0.5) with and without switchover delay. 

B. Motivation: Channels Without Memory 

In this section we assume that e ~ 0.5 so that the channel processes are i.i.d. over time. The stability region of the 
corresponding system with no-switchover time was established in 1401 : Ai, A2 G [0, 0.5] and Ai + A2 < 0.75. Note that when 
the switchover time is zero, the stability region is the same for both i.i.d. and Markovian channels, which is a special case of 
the results in ll32l . However, when the switchover time is non-zero, the stability region is reduced considerably: 

Theorem 1: The stability region of the system with i.i.d. channels and one-slot switchover delay is given by, 

A = {(Ai,A2)|Ai + A2 <0.5,Ai,A2 >0}. (1) 

In addition, the simple Exhaustive (Gated) policy is throughput- optimal. 

The proof is given in Appendix A for a more general system. The basic idea behind the proof is that as soon as the server 
switches to queue i under some policy, the time to the ON state is a geometric random variable with mean 2 slots, independent 
of the policy. Therefore, a necessary condition for stability is given by the stability condition for a system without switchover 
times and i.i.d. service times with geometric distribution of mean 2 slots as given by ^. The fact that the simple Gated policy 
is throughput-optimal follows from the observation that as the arrival rates are close to the boundary of the stability region, 
the fraction of time the server spends receiving packets dominates the fraction of time spent on switching B2l . 

As depicted in Fig. [3] the stability region of the system is considerably reduced for nonzero switchover delay. Note that 
for systems in which channels are always connected, the stability region is given by Ai + A2 < l,Ai,A2 > and is not 
affected by the switchover delay 1421 . Therefore, it is the combination of switchover delay and random connectivity that result 
in fundamental changes in system stability. 

Remark 1: As shown in Appendix A, the results in this subsection can easily be generalized to the case of non-symmetric 
Gilbert-Elliot channels with arbitrary switchover delays. For a system of 2 queues with arbitrary switchover delays and i.i.d. 
channels with probabilities p^, i G {1, 2}, A « the set of all A > such that Xi/pi + X2/P2 < 1- Moreover, simple Exhaustive 
(Gated) policy is throughput-optimal. 

When channel processes have memory, it is clear that one can achieve better throughput region than the i.i.d. channels case 
if the channels are positively correlated over time. This is because we can exploit the channel diversity when the channel states 
stay the same with high probability. In the following, we show that indeed the throughput region approaches the throughput 
region of no switchover time case in in B0I as the channels become more correlated over time. Note that the throughput 
region in BOl is the same for both i.i.d. and Markovian channels under the condition that probability of ON state for the i.i.d. 
channels is the same as the steady state probability of ON state for the two state Markovian channels. This fact can be derived 
as a special case of the seminal work of Neely in ll32l . 

C. Channels With Memory - Stability Region 

When switchover times are non-zero, the memory in the channel can be exploited to improve the stability region considerably. 
Moreover, as e — > 0, the stability region tends to that achieved by the system with no-switchover time and for < e < 0.5 it 
lies between the stability regions corresponding to the two extreme cases e = 0.5 and e — > as shown in Fig. |3] 



We start by analyzing the corresponding system with saturated queues, i.e., both queues are always non-empty. Let A^ 
denote the set of all time average expected departure rates that can be obtained from the two queues in the saturated system 
under all possible policies that are possibly history dependent, randomized and non-stationary. We will show that A = A^. 
We prove the necessary stability conditions in the following Lemma and establish sufficiency in the next section. 

Lemma 1: We have that 

AC A,. 

Proof: Given a policy tt for the dynamic queueing system specifying the switch and stay actions based possibly on 
observed channel and queue state information, consider the saturated system with the same sample path of channel realizations 
for t G {0,1,2,...} and the same set of actions as policy w at each time slot t e {0,1,2,...}. Let this policy for the saturated 
system be tt', and note that the policy n' can be non-stationarjO Let Di{t),i G {1,2} be total number departures by time t 
from queue-i in the original system under policy tt and let D^{t),i G {1,2} be the corresponding quantity for the saturated 
system under policy tt'. It is clear that \imt^oo{Di{t) + D2{t))/t < 1, where the same statement also holds for the limit of 
D[{t), i e {1, 2}. Since some of the ON channel states are wasted in the original system due to empty queues, we have 

Di{t)<D[{t), and, D2{t) < D'^it). (2) 

Therefore, the time average expectation of Di{t),i € {1,2} is also less than or equal to the time average expectation of 
D^{t), i e {1, 2}. This completes the proof since (|2]i holds under any policy it for the original system. ■ 

Now, we establish the region A^ by formulating the system dynamics as a Markov Decision Process (MDP). Let Sf = 
(7n(i), Ci(i), C2(i)) G S denote the system state at time t where S is the set of all states. Also, let at E A = {0, 1} denote the 
action taken at time slot t where A is the set of all actions at each state. Let H(t) = [st-JJ^^q U [ar]|^^o denote the full history 
of the system state until time t and let T(^) denote the set of all probability distributions on A. For the saturated system, a 
policy is a mapping from the set of all possible past histories to T(^) |l6l, ll27l . A policy is said to be stationary if, given 
a particular state, it applies the same decision rule in all stages and under a stationary policy, the process {st; < e N U {0}} 
forms a Markov chain. In each time slot t, the server observes the current state St and chooses an action at- Then the next 
state j is realized according to the transition probabilities P(j|s, a), which depend on the random channel processes. Now, we 
define the reward functions as follows: 

ri(s,a) = l if s= (1,1,1) or s = (1,1,0), and a = l (3) 

r2(s,a) = l if s= (2,1,1) or s = (2,0,1), and a = l, (4) 

and ri(s, a) = r2(s, a) = otherwise. That is, a reward is obtained when the server stays at an ON channel. We are interested 
in the set of all possible time average expected departure rates, therefore, given some ai,a2 > 0, ai + a2 — 1, we define the 
system reward at time t by r(s, a) = airi(s, a) + a2r2{s, a). The average reward of policy tt is defined as follows: 

1 ^ 

r^ =\imsup-E{j2rist,a^)}. 

Given some ai , 02 > 0, we are interested in the policy that achieves the maximum time average expected reward r* = max^ r'^ . 
This optimization problem is a discrete time MDP characterized by the state transition probabilities P(j|s, a) with 8 states 
and 2 actions per state. Furthermore, any given pair of states are accessible from each other (i.e., there exists a positive 
probability path between the states) under some stationary-deterministic policy. Therefore this MDP belongs to the class of 
Weakly Communicating MDP^ II35I . 

1) The State-Action Frequency Approach: For Weakly Communicating MDPs with finite state and action spaces and bounded 
rewards, there exists an optimal stationary-deterministic policy, given as a solution to standard Bellman's equation, with optimal 
average reward independent of the initial state ||35] Theorem 8.4.5]. This is because if a stationary policy has a nonconstant 
gain over initial states, one can construct another stationary policy with constant gain which dominates the former policy, 
which is possible since there exists a positive probability path between any two recurrent states under some stationary policy 
1271 . The state-action frequency approach, or the Dual Linear Program (LP) approach, given below provides a systematic and 
intuitive framework to solve such average cost MDPs, and it can be derived using Bellman's equation and the monotonicity 
property of Dynamic Programs [Section 8.8] 135 1 : 






Maximize y^ y^r{s,a)x{s,a) (5) 



^The policy n' can be based on keeping virtual queue length information, i.e., the queue lenghts in the dynamic queueing system. 

^In fact, other than the trivial suboptimal policy tTs that decides to stay with the current queue in all states, all stationary deterministic policies are unichain, 
namely, they have a single recurrent class regardless of the initial state. Hence, when tt^ is excluded, we have a Unichain MDP. 



subject to the balance equations 

x(s; 1) + x(s; 0) =^1 11^(^1^'' «)^(^'' «)> V s G 5, (6) 



s'es aeA 
the normaHzation condition 



_x(s;l)+x(s;0) = l, (7) 

sGS 

and the nonnegativity constraints 



sGS 



x(s, a) > 0, seS,ae A. (8) 

The feasible region of this LP constitutes a polytope called the state-action polytope X and the elements of this polytope 
X G X are called state-action frequency vectors. Clearly, X is a convex, bounded and closed set. Note that x(s; 1) can be 
interpreted as the stationary probability that action stay is taken at state s. More precisely, a point x e X corresponds to a 
stationary randomized policy that takes action a G {0, 1} at state s w.p. 

x( s a] 

Vi action a at state s) = ^ — '■ — ; -, a G A, s S S'^;, (9) 

x(s; 1) +x(s;0) 

where Sx is the set of recurrent states given by ^a; = {s e 5 : x(s; 1) + x(s; 0) > 0}, and actions are arbitrary for transient 
states s e S/Sx 1121, 1351 . 

Next we argue that the empirical state-action frequencies corresponding to any given policy (possibly randomized, non- 
stationary, or non-Markovian) lies in the state-action polytope X. This ensures us that the optimal solution to the dual LP 
in (|5]l is over possibly non-stationary and history-dependent policies. In the following we give the precise definition and the 
properties of the set of empirical state-action frequencies. We define the empirical state-action frequencies x^(s,a) as 

1 ^ 
i^(s,a) = ^51^{«'=«.''t=a}, (10) 

t=i 

where Ie is the indicator function of an event E, i.e., /^ = 1 if £' occurs and /^ = otherwise. Given a policy tt, let P"^ 
be the state-transition probabilities under the policy tt and (p — {(f)s) an initial state distribution with J^ses'^^ ^ ^- ^^ ^^'■ 
x^ As, a) be the expected empirical state-action frequencies under policy n and initial state distribution 0: 



1 ^ 

7^ X! ^^^'P'^i^t ^s,at = a\sa = s'). 



t=i s'es 

We let X7r,0 G T{S x A) (as in l27l . Il35l ) be the limiting expected state-action frequency vector, if it exists, starting from an 
initial state distribution (p, under a general policy tv (possibly randomized, non-stationary, or non-Markovian): 

a;,r,0(s,a)= limx^^(s,a). (11) 

Let the set of all limit points be defined by 

X.fi = {x e T{S X A) : there exists a policy tt s.t. 
the limit in ( fTTl i exists and x = X7r,0 }. 

Similarly let X^, denote the set of all limit points of a particular class of policies n', starting from an initial state distribution 
(f>. We let HsD denote the set of all stationary-deterministic policies and we let co{E) denote the closed convex hull of set S. 
The following theorem establishes the equivalency between the set of all achievable limiting state-action frequencies and the 
state-action polytope: 

Theorem 2: l35l Theorem 8.9.3], ll27l Theorem 3.1]. For any initial state distribution (p 

co(X^,J=X^=X. 

We have co(Xp^g^) C XjJ since convex combinations of vectors in Xj^ correspond to limiting expected state-action 
frequencies for stationary-randomized policies, which can also be obtained by time-sharing between stationary-deterministic 
policies. The inverse relation co{'K.'^ ) D Xj^ holds since for weakly communicating MDPs, there exists a stationary- 
deterministic optimal policy independent of the initial state distribution. Next, for any stationary-deterministic policy, the 



underlying Markov chain is stationary and therefore the limits X7r,0 exists and satisfies the constraints (|6]l, (|7j and ^ of 
the polytope X. Using co(Xf^g^) = Xj^ and the convexity of X establishes XjJ C X. Furthermore, via (|9}, every x G X 
corresponds to a stationary-randomized policy for which the limits x^r.^ exists, establishing X^ D X. 

Letting ext(X) denote the set of extreme (comer) points of X, an immediate corollary to Theorem |2] is as follows: 

Corollary 1: 1271 . M35II . For any initial state distribution (p 

ext(X) = X^^^. 

The intuition behind this corollary is that if x is a corner point of X, it cannot be expressed as a convex combination of any 
two other elements in X, therefore, for each state s only one action has a nonzero probability. 

Finally, we have that under any policy the probability of a large distance between the empirical expected state-action 
frequency vectors and the state-action polytope X decays exponentially fast in time. This result is similar to the mixing time 
of an underlying Markov chain to its steady state and we utilize such convergence results within the Lyapunov drift analysis 
for the dynamic queuing system in Section III-DI 

2) The Rate Polytope A,,.- Using the theory on state-action poly topes in the previous section, we characterize the set of 
all achievable time-average expected rates in the saturated system. A.,. The following linear transformation of the state-action 
polytope X defines the 2 dimensional rate polytope 1271 : 



A,, = 



|(?'i,?'2)| ri =^^x(s,a)ri(s,a) 

s&S a&A 

X! X! ^(^' «)^2(s, a), X e X|, 



where ri(s, a) and r2(s, a) are the reward functions defined in Q and (|4). This polytope is the set of all time average expected 
departure rate pairs that can be obtained in the saturated system, i.e., it is the rate region A^. An explicit way of characterizing 
As is given in Algorithm [T| 

Algorithm 1 Stability Region Characterization 
1: Given ai,a2>0, ai + a2 = 1 solve the following Linear Program (LP) 

max. airi(x) + a2?'2(x) 

subject to X e X. (12) 

2: For a given a2/ai ratio, there exists an optimal solution (r*,r2) of the LP in ( fT2b at a corner point of Ag. Find all 
possible corner points and take their convex combination. 

The fundamental theorem of Linear Programming guarantees that an optimal solution of the LP in (fT2l i lies at a corner 
(extreme) point of the polytope X ifTOI . Furthermore, the one-to-one correspondence between the extreme points of the polytope 
X and stationary-deterministic polices stated in Corollary [T] is useful for finding the solutions of the above LP for all possible 
a2/ai ratios. Namely, there are a total of 2^ stationary-deterministic policies since we have 8 states and 2 actions per state 
and finding the rate pairs corresponding to these 256 stationary-deterministic policies and taking their convex combination 
gives A,. Fortunately, we do not have to go through this tedious procedure. The fact that at a vertex of (fT2l i either x(s; 1) 
or x(s; 0) has to be zero for each s e S provides a useful guideline for analytically solving this LP. The following theorem 
characterizes the stability region explicitly. It shows that the stability region enlarges as the channel has more memory and that 
there is a critical value of the channel correlation parameter given hy ec = 1 — \/2/2 at which the structure of the stability 
region changes. 

Theorem 3: The rate region As is the set of all rates ri > 0, r2 > that for t < tc satisfy 
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Fig. 4: Stability region under channels with memory, with and without switchover delay for (a) e = 0.25 < Cc and (b) e = 0.40 > Cc- 



and for e > Cc satisfy 



ri + (l-e)(3-2e)r2 < 



(l-6)(3-2e) 



ri +r2 < T - 7: 



(l-e)(3-2e)ri+r2 



< 



3 _ £ 

4 ~ 2 
(l-6)(3-2e) 



The proof of the theorem is given in Appendix B and it is based on solving the LP in (T% for all weights ai and 02 to find 
the corner points of As, and then applying Algorithm [T] The following observation follows from Theorem |3] 

Observation 1: The maximum achievable sum-rate in the saturated system is given by 

3 e 

4 ~ 2' 



ri +r2 



Note that ri + r2 < | is the boundary of the stability region for the system without switchover delay analyzed in POl . where 
the probability that at least 1 channel is in ON state is 3/4. Therefore, e/2 is the throughput loss due to the 1 slot switchover 
delay. This throughput loss corresponds to the probability that the server is at a queue with an OFF state when the other queue 
is in an ON state. 

The stability regions for the two ranges of e are displayed in Fig. |4] (a) and (b). As e ^- 0.5, the stability region converges 
to that of the i.i.d. channels with ON probability equal to 0.5. In this regime, knowledge of the current channel state is of 
no value. As e — >■ the stability region converges to that for the system with no-switchover time in BOl . In this regime, the 
channels are likely to stay the same for many consecutive time slots, therefore, the effect of switching delay is negligible. 

The rate region Ag for the case of non-symmetric Gilbert-Elliot channels is given in Appendix B. 

Remark 2: The stability region characterization in terms of state-action frequencies is general. For instance, this technique 
can be used to establish the stability regions of systems with more than two queues, arbitrary switchover times, and more 
complicated Markovian channel processes. Of course, explicit characterization as in Theorem |3] may not always be possible. 

D. Frame Based Dynamic Control (FBDC) Policy 

We propose a frame-based dynamic control (FBDC) policy inspired by the characterization of the stability region in terms 
of state-action frequencies and prove that it is throughput-optimal asymptotically in the frame length. The motivation behind 
the FBDC policy is that a policy tt* that achieves the optimization in ( fT2] i for given weights ai and ^2 for the saturated 
system should achieve a good performance in the original system when the queue sizes Qi and Q2 are used as weights. This 
is because first, the policy tt* will lead to similar average departure rates in both systems for sufficiently high queue sizes, and 
second, the usage of queue sizes as weights creates self adjusting policies that capture the dynamic changes due to stochastic 
arrivals similar to Max-Weight scheduling in fW\. Specifically, we divide the time into equal-size intervals of T slots and let 
QiiJT) and Q2{jT) be the queue lengths at the beginning of the jth interval. We find the deterministic policy that optimally 
solves ( fT2] i when Qi{jT) and Q2{jT) are used as weights and then apply this policy in each time slot of the frame. The 
FBDC policy is described in Algorithm |2] in details. 

There exists an optimal solution {rl , rj) of the LP in ( fT3] ) that is a comer point of Ag |TOl and the policy tt* that corresponds 
to this point is a stationary-deterministic policy by Corollary [T] 

Theorem 4: For any 6 > 0, there exists a large enough frame length T such that the FBDC policy stabilizes the system for 
all arrival rates within the S-stripped stability region Af = A^ — 51. 



Algorithm 2 Frame Based Dynamic Control (FBDC) Policy 



1: Find the policy tt* that optimally solves the following LP 

max.{rj^r,} Qi{jT)ri + Q2(jT)r2 
subject to iri,r2) e As 

where A, is the rate polytope derived in Section III-CI 
2: Apply TV* in each time slot of the frame. 
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TABLE I: FBDC policy mapping from the queue sizes to the comers of Ag , 69 , 61 , 62 1 f'3 1 
{m(t),Ci{t),C2{t)) the optimal action is specified. The thresholds on Q2/Q1 are 0, T* = 



64,65 shown in Fig. |4] (a), for e < e^- For each state 
e/(l - ef,T* = (1 - e)/(l + e - e^), 1, T^ = (1 + 



An immediate corollary to this theorem is as follows: 

Corollary 2: The FBDC policy is throughput-optimal asymptotically in the frame length. 
The proof of Theorem |4] is given in Appendix D. It performs a drift analysis using the standard quadratic Lyapunov function. 
However, it is novel in utilizing the state-action frequency framework of MDP theory within the Lyapunov drift arguments. The 
basic idea is that, for sufficiently large queue lengths, when the optimal policy solving ( fT3] l. tt*, is applied over a sufficiently 
long frame of T slots, the average output rates of both the actual system and the corresponding saturated system converge to r*. 
For the saturated system, the probability of a large difference between empirical and steady state rates decreases exponentially 
fast in T IZTl . similar to the convergence of a positive recurrent Markov chain to its steady state. Therefore, for sufficiently 
large queue lengths, the difference between the empirical rates in the actual system and r* also decreases with T. This 
ultimately results in a negative Lyapunov drift when A is inside the (5(T)-stripped stability region since from (fT?t we have 
QiiJTVi + Q2{iT)rl > Qi(jT)Ai + Q2{jT)X2. 

The FBDC policy is easy to implement since it does not require the arrival rate information for stabilizing the system for 
arrival rates in A — d{T)l, and it does not require the solution of the LP ([13) for each frame. Instead, one can solve the LP (fT3b 
for all possible (Qi, Q2) pairs only once in advance and create a mapping from {Qi, Q2) pairs to the comers of the stability 
region. Then, this mapping can be used to find the corresponding optimal saturated-system policy to be applied during each 
frame. Solving the LP in (fTSl l for all possible (Qi, Q2) pairs is possible because first, the solution of the LP will be one of 
the corner points of the stability region in Fig. ID and second, the weights {Qi, Q2), which are the inputs to the LP, determine 
which corner point is optimal. The theory of Linear Programming suggests that the solution to the LP in ( fT3l ) depends only 
on the relative value of the weights (Qi, Q2) with respect to each other Namely, changing the queue size ratio Q2/Q1 varies 
the slope of the objective function of the LP in ( fTSl l. and the value of this slope Q2/Q1 with respect to the slopes of the lines 
in the stability region in Fig. |4] determine which corner point the FBDC policy operates on. These mappings from the queue 
size ratios to the corners of the stability region are shown in Table I] for the case of e < Ec and in Table HI] for the case of 
e > €c. The corresponding mappings for the FBDC policy for the case of non-symmetric Gilbert-Elliot channels are shown in 
Appendix B. Given these tables, one no longer needs to solve the LP (fTSl l for each frame, but just has to perform a simple 
table look-up to determine the optimal policy to use in each frame. 

s 

In the next subsection we provide an upper bound to the long-run packet-average delay under the FBDC policy, which is 
linear in T. This suggest that the packet delay increases with increasing frame lengths as expected. However, such increases are 
at most linear in T. Note that the FBDC policy can also be implemented without any frames by setting T = 1, i.e., by solving 
the LP in Algorithm |2] in each time slot. The simulation results in Section |IV] suggest that the FBDC policy implemented 
without frames has a similar throughput performance to the original FBDC policy. This is because for large queue lengths, the 
optimal solution of the LP in ( fT3] ) depends on the queue length ratios, and hence, the policy tt* that solves the LP optimally 
does not change fast when the queue lengths get large. When the policy is implemented without the use of frames, it becomes 
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TABLE II: FBDC policy mapping from the queue sizes to the comers of As, bo , bi , ^2 1 ''s shown in Fig. |4] (b), for t > Ec- For each state 
s = {m{t), Ci{t), (72 (t)) the optimal action is specified. The thresholds on Q2/Q1 are 0, T* = 1/((1 - £){3 ~ 2e)), 1, Tj* = (1 - €){3 - 2e). 

more adaptive to dynamic changes in the queue lengths, which resuhs in a better delay performance than the frame-based 
implementations . 

1) Delay Upper Bound: The delay upper bound in this section is easily derived once the stability of the FBDC algorithm 
is established. The stability proof utilizes the following quadratic Lyapunov function 



L{Q{t))=J2QUt). 



i=l 

which represents a quadratic measure of the total load in the system at time slot t. Let tk denote the time slots at the frame 
boundaries, fc = 0, 1, ..., and define the T-step conditional drift 

Aritk) = E [L{Q{tk + T)) - L{Q{tk))\Qitk)] , 
The following drift expression follows from the stability analysis in Appendix C: 

AT(tfe) 



2T 



<BT-(^g,(i,.)U 



where B = 1 + A'^^^^, A is strictly inside the 5-stripped stability region A — 51, and £, > Q represents a measure of the 
distance of A to the boundary of A — 51. Taking expectations with respect to Q(tA;), writing a similar expression over the 
frame boundaries tk,k £ {0, 1, 2, ..., K}, summing them and telescoping these expressions lead to 



K-l 



L{Q{tK)) - i(Q(0)) < 2KBT^ - 2eT^E 
Using L(Q(ix)) > and i(Q(0)) = 0, we have 



fc=0 



Eq»(^^) 



K-1 



1 RT^ 

limsup- Y, 5Z^[^'(*fc)] ^ -Y' 



k=0 i 



^T-1 



For t e itk,tk+i) we have Q,{t) < Q^{tk) + ErJ Mtk+r). Therefore, E[Q,{t)] < E[Q,{tk)]+TX, < E[Q,{tk)]+TA„ 
Therefore, for Tk = KT we have 



KT-l 



limsup -i- J2 J2^1QM 

<limsup -i-^^TE[Q.(tfe)]+r2A„,,< i:?±^|2^. 



fe=0 I 

Dividing by the total arrival rate into the system J^i ^i ^"d applying Little's law, the average delay is upper bounded by an 
expression that is linear in the frame length T. 

In the next section we consider Myopic policies that do not require the solution of an LP and that are able to stabilize the 
network for arrival rates within over 90% of the stability region. Simulation results in Section |IV] suggest that the Myopic 
policies may in fact achieve the full stability region while providing better delay performance than the FBDC policy for most 
arrival rates. 
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E. Myopic Control Policies 

We investigate the performance of simple Myopic policies that make scheduling/switching decisions according to weight 
functions that are products of the queue lengths and the channel predictions for a small number of slots into the future. We 
refer to a Myopic policy considering k future time slots as the fc-Lookahead Myopic policy. We implement these policies 
over frames of length T time slots where during the jth frame, the queue lengths at the beginning of the frame, Qi{jT) and 
Q2{jT), are used for weight calculations during the frame. Specifically, in the 1-Lookahead Myopic policy, assuming that the 
server is with queue 1 at some t E {jT, ...,j{T + 1) — 1}, the weight of queue 1 is the product of Qi{jT) and the summation 
of the current state of the channel process Ci and the probability that Ci will be in the ON state at t + 1. The weight of 
queue 2 is calculated similarly, however, the current state of the channel process C2 is not included in the weight since queue 
2 is not available to the server in the current time slot. The detailed description of the 1-Lookahead Myopic policy is given in 
Algorithm [3] below. 

Algorithm 3 1-Lookahead Myopic Policy 

I: Assuming that the server is currently with queue 1 and the system is at the jth frame, calculate the following weights in 
each time slot of the current frame; 

Wi{t) = Q,{jT)(Ciit)+E[Ci{t + l)\Ci{t)]^ 

W2{t) = Q2{jT)E[C2{t + l)|C2(t)] . (14) 

2: If Wi{t) > W2{t) Stay with queue 1, otherwise, switch to the other queue. A similar rule applies for queue 2. 

Next, we establish a lower bound on the stability region of the 1-Lookahead Myopic Policy by comparing its drift over a 
frame to the drift of the FBDC policy. 

Theorem 5: The 1-Lookahead Myopic policy achieves at least 'y -fraction of the stability region Ag asymptotically in T 
where 7 > 90%. 

The proof is constructive and will be establish in various steps in the following. The basic idea behind the proof is that the 
1-Lookahead Myopic (OLM) policy produces a mapping from the set of queue sizes to the stationary deterministic policies 
corresponding to the comers of the stability region. This mapping is similar to that of the FBDC policy, however, the thresholds 
on the queue size ratios Q2/Q1 are determined according to (flTt : 

Mapping from queue sizes to actions. Case-1: e < ec 

For e < Cc, there are 6 corners in the stability region denoted by bo, &i, ..., 65 where 60 is (0, 0.5) and 65 is (0.5, 0) as shown 

in Fig. |4](a). We derive conditions on Q2/Q1 such that the OLM policy chooses the stationary deterministic decisions that 

correspond to a given corner point. 

Corner bo: 

Optimal actions are to stay at queue-2 for every channel condition. Therefore, the server chooses queue-2 even when the channel 

state is Ci{t),C2{t) = (1,0). Therefore, using (fl4t . for the Myopic policy to take the deterministic actions corresponding to 

&o we need 

Qi.{l - e) < Q2.{e) ^7^>^—^- 

This means that if we apply the Myopic policy with coefficients Qi, Q2 such that Q2/Q1 > (1 — e)/e, then the system output 

rate will be driven towards the comer point 60 (both in the saturated system or in the actual system with large enough arrival 

rates). 

Corner bi. 

The optimal actions for the corner point 61 are as follows; At queue-1, for the channel state 10:stay, for the channel states 11, 

01 and 00: switch. At queue-2, for the channel state 10; switch, for the channel states 11, 01 and 00: stay. The most limiting 

conditions are 11 at queue-1 and 10 at queue-2. Therefore we need, Qi(2 — e) < Q2(l — e) and Qi{l — e) > Q2i- Combining 

these we have 

2-e Q2 l__e 

Note that the condition e < Cc = 1 - \/2/2 implies that ^ > fff. 

Corner 62: 

The optimal actions for the comer point bi are as follows: At queue-1, for the channel state 10 and ll:stay, for the channel 

states 01 and 00: switch. At queue-2, for the channel states 10: switch, for the channel states 11, 01 and 00: stay. The most 
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TABLE III: 1 Lookahead Myopic policy mapping from the queue sizes to the comers of As, feo i ''1 1 ''2 1 ''3 1 ''4 1 t's shown in Fig. |4] (a), for e < ec- For 
each state s = (m(t), Ci(t), C2(i)) the optimal action is specified. The thresholds on Q2/Q1 are 0, Ti = e/{l - e),T2 = (1 - e)/(2 - e),l,Ta = 
{2 — e)/{l — e),T4 = (1 — e)/e. The corresponding thresholds for the FBDC policy are 0, T^ ,T2 , 1, T3 , T^ . For example, comer 62 is chosen in the FBDC 
policy if 1 < Q2/Q1 < Tg*, whereas in the OLM policy if 1 < Q2/Q1 < T^. 

limiting conditions are 11 at queue-1 and 00. Therefore we need, Qi(2 — e) > (52(1 ~ e) and Qi < Q2- Combining these we 
have 

Qi 1 -e 
The conditions for the rest of the comers are symmetric and can be found similarly to obtain the mapping in Fig. |llll 

Mapping from queue sizes to actions. Case-2: e > ec 

In this case there are 4 comer points in the throughput region. We enumerate these comers as fooi ^2, ^3, ^5 where 60 is (0, 0.5) 

and 65 is (0.5,0). 

Corner 60: 

The analysis is the same as the 60 analysis in the previous case and we obtain that for the Myopic policy to take the deterministic 

actions corresponding to feo we need 

Q2 1 - £ 

Oi e ■ 

Corner 62: 

This is the same corner point as in the previous case corresponding to the same deterministic policy: At queue-1, for the 
channel state 10 and ll:stay, for the channel states 01 and 00: switch. At queue-2, for the channel states 10: switch, for the 
channel states 11, 01 and 00: stay. The most limiting conditions are 10 at queue-2 (since e > Cc we have ^-^ < jE^) and 00. 
Therefore we need, Qi{l — e) > Q2e and Qi < Q2- Combining these we have 

, Q2 l-e 
1< — < . 

Qi e 

The conditions for the rest of the comers are symmetric and can be found similarly to obtain the mapping in Fig. |lV]for e > ec- 



The conditions for the comers 62 and 63 are symmetric, completing the mapping from the queue sizes to the comers of Ag 
for e > ec shown in Table HV] This mapping is in general different from the corresponding mapping of the FBDC policy in 
Table HIl Therefore, for a given ratio of the queue sizes Q2/Q1, the FBDC and the OLM policies may apply different stationary 
deterministic policies corresponding to different comer points of As, denoted by r* and f respectively. The shaded intervals 
of Q2/Q1 in Table HVl are the intervals in which the OLM and the FBDC policies apply different policies. A similar mapping 
can be obtained for the OLM policy for e < ec. The corresponding mapping for the OLM policy for the case of non-symmetric 
Gilbert-Elliot channels is given in Appendix B. 

The following lemma is proved in Appendix E and completes the proof by establishing the 90% bound on the weighted 
average departure rate of the OLM policy w.r.t. to that of the FBDC policy. 

Lemma 2: We have that 



* 



> 90%. 



(15) 



Furthermore, $ > 90% is a sufficient condition for the OLM policy to achieve at least 90% of As asymptotically in T. 

A similar analysis shows that the 2-Lookahead Myopic Policy achieves at least 94% of A^, while the 3 -Lookahead Myopic 
Policy achieves at least 96% of As. The fc-Lookahead Myopic Policy is the same as before except that the following weight 
functions are used for scheduling decisions: Assuming the server is with queue 1 at time slot t, 

Wi{t) = Qi(jT)(Ci(t) + E'=i]E[Ci(< + T)|Ci(f)]) and W2{t) = Q2(jT)E!:=iIE[C2(t + T)|C2(t)]. 
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TABLE IV: 1 Lookahead Myopic policy mapping from the queue sizes to the comers of As, boi ^11^21^3 shown in Fig. |4] (b), for e > tc- For each 
state s = (m{t),Ci{t),C2{t)) the optimal action is specified. The thresholds on Q2/Q1 are 0, Ti = e/(l — e),l,T2 = (1 — e)/e. The con'esponding 



thresholds for the FBDC policy are 0, T* , 
if 1 < Q2/Q1 < T2. 



1,T2 . For example, comer 61 is chosen in the FBDC policy if 1 < Q2/Q1 < T2 , whereas in the OLM policy 



III. General System 

In this section we extend the results developed in the previous section to the general case of an arbitrary number of queues 
in the system. 

A. Model 

Consider the same model as in Section Hl-AI with A^ > 1 queues for some iV G N as shown in Fig. [T] Let the i.i.d. process 
Ai{t) with arrival rate Xi denote the number of arrivals to queue i at time slot t, where E[Af (t)] < A^^ax' * ^ l^' ^i ••■, ^}- Let 
Ci{t) be the channel (connectivity) process of queue i, i £ {1, 2, ..., N}, that forms the two-state Markov chain with transition 
probabilities poi and pio as shown in Fig. [2] We assume that the processes Ai{t),i G {1, •••, N} and Ci{t),i £ {1, ..., N} are 
independent. It takes one slot for the server to switch from one queue to the other, and m{t) E {1, •••, N} denotes the queue 
at which the server is present at slot t. Let Sj = (m{t), Ci(t), ..., Cf^{t)) e S denote the state of the corresponding saturated 
system at time t where S is the set of all states. The action a{t) in each time slot is to choose the queue at which the server 
will be present in the next time slot, i.e., at G {1, •••, N} = A where A is the set of all actions at each state. 

B. Stability Region 

In this section we characterize the stability region of the general system under non-symmetric channel modelo For the 
case of i.i.d. channel processes we explicitly characterize the stability region and the throughput-optimal policy. For Markovian 
channel models, we extend the stability region characterization in terms of state-action frequencies to the general system. 
Furthermore, we develop a tight outer bound on the stability region using an upper bound on the sum-throughput and show 
that a simple myopic policy achieves this upper bound for the corresponding saturated system. 

A dynamic server allocation problem over parallel channels with randomly varying connectivity and limited channel sensing 
has been investigated in H], ||2l, 1471 under the Gilbert-Elliot channel model. The goal in H], 121, BTl is to maximize the 
sum-rate for the saturated system, where it is proved that a myopic policy is optimal. In this section we prove that a myopic 
policy is sum-rate optimal under the Gilbert-Elliot channel model and 1-slot server switchover delay. Furthermore, our goal is 
to characterize the set of all achievable rates, i.e., the stability region, together with a throughput-optimal scheduling algorithm 
for the dynamic queuing system. 

1) Memoryless Channels: The results established in Section Hl-BI for the case of i.i.d. connectivity processes can easily be 
extended to the system of N queues with non-symmetric i.i.d. channels as the same intuition applies for the general case. We 
state this result in the following theorem whose proof can be found in Appendix A. 

Theorem 6: For a system of N queues with arbitrary switchover times and i.i.d. channels with probabilities pi,i G {1, ..., N}, 
the stability region A is given by 

In addition, the simple Exhaustive (Gated) policy is throughput-optimal. 

As for the case of two queues, the simultaneous presence of randomly varying connectivity and the switchover delay significantly 
reduces the stability region as compared to the corresponding system without switchover delay analyzed in ||40l . Furthermore, 
when the channel processes are memoryless, no policy can take advantage of the channel diversity as the simple queue-bUnd 
Exhaustive- type policies are throughput-optimal. 

''For Markovian (Gilbert-Elliot) channels, we preserve the symmetry of the channel processes across the queues. 
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In the next section, we show that, similar to the case of two queues, the memory in the channel improves the stability region 
of the general system. 

2) Channels With Memory: Similar to Section III-CI we start by establishing the rate region As by formulating an MDP for 
rate maximization in the corresponding saturated system. The reward functions in this case are given as follows; 

ri(s, a) = 1 if m = i, Ci = 1, and a~i, i = 1, ..., N, (16) 

and ri{s, a) = otherwise, where m denotes the queue at which the server is present. That is, one reward is obtained when 
the server stays at a queue with an ON channel. Given some ai > 0, i E {1, ■.-, N}, J^i "^j = 1' we define the system reward 
at time t as 

N 

r{s,a) = y^^airi{s,a). 
The average reward of policy tt is defined as 

t=i 
Therefore, the problem of maximizing the time average expected reward over all policies, r* = max^r'^, is a discrete 
time MDP characterized by the state transition probabilities P(s'|s, a) with N2^ states and N possible actions per state. 
Furthermore, similar to the two-queue system, there exists a positive probability path between any given pair of states under 
some stationary-deterministic policy. Therefore, this MDP belongs to the class of Weakly Communicating MDPs f35ll for which 
there exists a stationary-deterministic optimal policy independent of the initial state ||35]| . The state-action polytope, X is the 
set of state-action frequency vectors x that satisfy the balance equations 

^ x(s, a)^Y.Y. P(^l^'' «)^(^'' a), V s e 5, (17) 

the normalization condition 

^^x(s,a) = 1, 

and the nonnegativity constraints 

x(s, a) > 0, for s e 5, a e A, 

where the transition probabilities P(s|s',a) are functions of the channel parameters pio and poi- The following linear 
transformation of the state-action polytope X defines the rate polytope A,, namely, the set of all time average expected 
rate pairs that can be obtained in the saturated system. 

As=|r|ri=^^x(s,a)ri(s,a),xe X,ie {1, 2, ..., iV}|, 

where the reward functions ri(s, a),i € {1, ..., A^}, are defined in (fT6] l. Algorithm |4] gives an alternative characterization of 
the rate region A^. 

Algorithm 4 Stability Region Characterization 
1: Given ai, ..., aAr>0, '^^ a^ — 1, solve the following LP 

N 

max. y aivAyi) 

1=1 
subject to X e X. (18) 

2: There exists an optimal solution (r^, ...,r^) of this LP that lies at a corner point of Ag. Find all possible corner points 
and take their convex combination. 

Similar to the two-queue case, the fundamental theorem of Linear Programming guarantees existence of an optimal solution 
to ( fTsT i at a comer point of the polytope X ifTOl . We will establish in the next section that the rate region, A^ is in fact 
achievable in the dynamic queueing system, which will imply that A = A,,. For the case of 3 queues. Fig. |5] shows the 
stability region A. As expected, the stability region is significantly reduced as compared to the corresponding system with zero 
switchover delays analyzed in 001. 
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Fig. 5: Stability region for 3 parallel queues for pm = poi = 0-3- 




Fig. 6: Stability region outer bound for 3 parallel queues for pio = poi = 0.3. 



3) Analytical Outer Bound For The Stability Region: In this section we first derive an upper bound to the sum-throughput 
in the saturated system and then use it to characterize an outer bound to the rate region A^. Let Cq = -, — y-° sjv denote 
the probabihty that all channels are in OFF state in steady state. 

Lemma 3: An upper bound on the sum-rate in the saturated system is given by 



N 

E 

4=1 



n < 1 



cr-(pio(i-cr)-mcr)- 



(19) 



The proof is given in Appendix F. In the next section we propose a simple myopic policy for the saturated system that achieves 
this upper bound. Similar to the case of two-queues, the surface X]i=i i^i ^ ^ ^ Cq is one of the boundaries of the stability 
region for the system without switchover delay analyzed in BOl . where the probability that at least 1 channel is in ON state 



,{N) 



^(JV)^ 



,(W) 



in steady state is 1 — Cq" ' . Therefore, pio{l — Cq ') — poiC'o''^ is the throughput loss due to 1 slot switchover delay in 
our system. The analysis of the myopic policy in the next section shows that this throughput loss due to switchover delay 
corresponds to the probability that the server is at a queue with OFF state when at least one other queue is in ON state. For 
the case of A^ = 3 queues, the sum-throughput upper bound in Lemma |3] is the hexagonal region at the center of the plot in 

Fig. El 

Because any convex combination of ri,i e {1, ...,N}, must lie under the sum-rate surface, ([19) is in fact an outer bound 
on the whole rate region A,. Furthermore, no queue can achieve a time average expected rate that is greater than the steady 
state probability that the corresponding queue is in ON state, i.e., poi/(pio +Pm)- Therefore, the intersection of these A^ + 1 
surfaces in the N dimensional space constitutes an outer bound for the rate -region A,. Note that this outer bound is tight in 
that the sum-rate surface of the maximum rate region A^, as well as the corner points poi/{piQ +pai) coincide with the outer 
bound. This outer bound with respect to the rate region are displayed in Fig. |6]for the case of iV = 3 nodes. 
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C. Myopic Policy for the Saturated System 

We show in this section that a simple and intuitive policy, termed the Greedy Myopic (GM) policy, achieves the sum rate 
maximization for the saturated system. This policy is a greedy policy in that under the policy, if the current queue is available 
to serve, the server serves it. Otherwise, the server switches to a queue with ON channel state, if such a queue exists. The 
policy is described in Algorithm |5] Recall that m{t) denotes the queue the server is present at time slot t. 

Algorithm 5 Greedy Myopic Policy 

1: For all time slots t, if Cm(t){t) = 1' serve queue ?7i(i). 

2: Otherwise, if 3j £ {1, ...,N},j ^ m{t), such that Cj{t) = 1, among the queues that have ON channel state, switch to 
the queue with the smallest index in a cyclic order starting from queue m{t). 

The cyclic switching order under the GM policy is as follows: If the server is at queue i and the decision is to switch, then 
the server switches to queue j, where for i = N, j ~ argmiiij^ji _7v-i}(Cj(0 — 1) ^i^^ for i "^ N if 3j E {i + 1, ...,N}, 
such that Cj{t) = 1, we have j = argmin^g^i+i ...^jv}(Ci(^) = 1)> if not, then j = argminj£{i^..._j_i}(Cj(t) = 1). 

Theorem 7: The GM policy achieves the sum-rate upper bound. 

Proof: Given a fixed decision rule at each state, the system state forms a finite state space, irreducible and positive recurrent 
Markov chain. Therefore, under the GM policy, the system state converges to a steady state distribution. We partition the total 
probability space into three disjoint events: 
Ei: the event that all the channels are in OFF state, 

E2'. the event that at least 1 channel is in ON state and the server is at a queue with ON state 
£^3: the event that at least 1 channel is in ON state and the server is at a queue with OFF state 
Since these events are disjoint we have, 

l=¥{Ei)+]P{E2)+P{E3). 

We have P(£'i) ~ Cq by definition. Since the GM policy decides to serve the current queue if it is in ON state, P(£'2) 
gives the sum throughput J^i ^i for the GM policy. Therefore, we have that under the GM policy 






{N) 






n = l-C^"' -F{E3) 



We show that ^{E^) = piq{1 — Cq )~PoiCq . Consider a time slot t in steady state and let ^(i) be the number of channels 
with ON states at time slot t and let Eo (t) be the event that the server is at a queue with OFF state at time slot t. We have 

P(£'3)=P(£'o(i) and 1 < K(i) < iV-1) =P(SoW andK(i) > 1) 
= F{Eo{t) and K{t) > l\K{t-l) > l)P(K(i-l) > 1) 
+ P(£'o(t) and K{t) > l\K{t-l) = 0)F{K{t-l) = 0). 

Since i is a time slot in steady state, we have that 

¥{K{t - 1) = 0) := C^^\ Therefore, FiEs) is given by 

P {i^{t)>l\Eo{t),K{t-l)>l)F{Eo{t)\K{t-l)>l){l~do'') 
+F iKit)>l\Eo{t),Kit-l)=0)F{Eoit)\,i{t-l) = 0)C^^\ 

We have F{Eo{t)\K{t — I) > 1) = Pio since the GM policy chooses a queue with ON state if there is such a queue and 
F{Eo(t)\K{t—l) = 0) is the probability that the queue chosen by the GM policy keeps its OFF channel state, given by 1 —poi- 

F{E3) = (1 - F{K{t) = 0\Eo{t),K{t-l) > l))pio(l - O 
+ (1 - F{K{t) = 0\Eo{t),K{t-l)=0)){l - poi)do''^ 

n r(^\f^ n'^W=0,i?oW|K(t-l)>l 
= _Pio(i - C-o ) i-- 



+(i-poi)crMi- 



F{Eo{t)\Kit-l)>l) 
(TV) f, _ F{K{t)^0,Eo{t)\K{t-l) = 0) 
F{Eo{t)\K{t-l)=0) 



„ rWJ, F(K(t)=Q\n{t-l)>l) 

= Pio(l-Co 1 

V Pw 

^.1 ^^WA F{n{t)^Q\n{t-l)=Q) 
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We have that ¥{K{t) = 0\K{t-l) > 1) is given by 

P(K(t) = 0) - ¥{n{t) = 0\K{t - 1) = 0)P(K(t - 1) = 0) 
P(K(i-l) > 1) ' 

which is equivalent to (C^^^ - (1 - poi)^C^^V(l - C'o^^)- Therefore, P(£;3) is given by 

V pio(l-C^ ') 

+ (l-Poi)Co 1 ] — ;; — 

As mentioned in the previous section, P(£'3) is the throughput loss due to switching as it represents the fraction of time the 
server is at a queue with OFF state when there are queues with ON state in the system. ■ 

D. Frame-Based Dynamic Control Policy 

In this section we generalize the FBDC policy to the general system and show that it is throughput-optimal asymptotically 
in the frame length for the general case. The FBDC algorithm for the general system is very similar to the FBDC algorithm 
described for two queues in Section III-DI Specifically, the time is divided into equal-size intervals of T slots. We find the 
stationary-deterministic policy that optimally solves (fTSl l for the saturated system when Qi{jT), ..., Qn{JT) are used as weights 
and then apply this policy in each time slot of the frame in the actual system. The FBDC policy is described in Algorithm |6] 
in details. 

Algorithm 6 Frame Based Dynamic Control (FBDC) Policy 
1: Find the optimal solution to the following LP 

subject to r = (ri, ..., rjv) e A^ (20) 

where A,, is the rate region for the saturated system. 
2: The optimal solution (rj;, ...,r^) in step 1 is a comer point of A, that corresponds to a stationary-deterministic policy 
denoted by tt*. Apply tt* in each time slot of the frame. 

Theorem 8: For any S > 0, there exists a large enough frame length T such that the FBDC policy stabilizes the system for 
all arrival rates within the 5-stripped stability region Af = A^ — 51. 

The proof is very similar to the proof of Theorem 2]and is omitted. The theorem establishes the asymptotic throughput-optimality 
of the FBDC policy for the general system. 

Remark 3: The FBDC policy provides a new framework for developing throughput-optimal policies for network control. 
Namely, given any queuing system whose corresponding saturated system is Markovian with finite state and action spaces, 
throughput-optimality is achieved by solving an LP in order to find the stationary MDP solution of the corresponding saturated 
system and applying this solution over a frame in the actual system. In particular, the FBDC policy can stabilize systems with 
arbitrary switchover times and more complicated Markov modulated channel structures. The FBDC policy can also be used 
to achieve throughput-optimality for classical network control problems such as the parallel queueing systems in ll33l . 1401 . 
scheduling in switches in ll36ll or scheduling under delayed channel state information P6l . 

Similar to the delay analysis in Section III-Dll for the two-queue system, a delay upper bound that is linear in the frame 
length T can be obtained for the FBDC policy for the the general system. Moreover, the FBDC policy for the general system 
can also be implemented without any frames by setting T = 1, i.e., by solving the LP in Algorithm |6] in each time slot. The 
simulation results regarding such implementations suggest that the FBDC policy implemented without frames has a similar 
throughput performance and an improved delay performance as compared to the original FBDC policy. 

1) Discussion: For systems with switchover delay, it is well-known that the celebrated Max- Weight scheduling policy is 
not throughput-optimal lITTl . In the absence of randomly varying connectivity, variable frame based generalizations of the 
Max- Weight policy are throughput-optimal I TS] . However, when the switchover delay and randomly varying connectivity 
are simultaneously present in the system, the FBDC policy is the only poUcy to achieve throughput-optimaUty and it has a 
significantly different structure from the Max- Weight poUcy. 
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The FBDC policy for a fixed frame length T does not require the arrival rate information for stabilizing the system for 
arrival rates in A — (5(T)1, however, it requires the knowledge of the channel connectivity parameters picpoi- To deal with this 
problem one can estimate the channel parameters periodically and use these estimates to solve the LP in ( |20] |. This approach, 
of course, incurs a throughput loss depending on how large the estimation error is. 

As mentioned in Remark |3] the FBDC policy can stabilize a large class of network control problems whose corresponding 
saturated system is weakly communicating Markovian with a finite state and action spaces. However, one caveat of the FBDC 
policy is that the state space of the LP that needs to be solved increases exponentially with the number of links in the system. 
The celebrated Max-Weight policy (which is not stabilizing for the system considered here) has linear complexity for the single 
server system considered in this paper However, for general multi-server systems with N servers or for a single hop network 
with N interfering links, the Max- Weight poUcy has to solve a maximum-independent set problem over all links at each time 
slot, which is a hard problem whose state space is also exponential in the number of links A^. The FBDC policy on the other 
hand, only has to solve an LP, for which there are standard solvers available such as CPLEX. Furthermore, the FBDC policy 
has to solve the LP once per frame, whereas the the Max- Weight policy performs maximum-independent set computation each 
time slot. If the frame length for the FBDC policy is chosen to be bigger than the computational complexity of the LP in 
( l20t . then the per-slot computational complexity of the algorithm is reduced to 0(1). Such a frame -based implementation is 
also possible for the Max- Weight policy to reduce its complexity to 0(1) per time slot. On the other hand, the shortcoming 
of such an approach for both policies is the increase in delay as a result of the larger frame length. This outlines a tradeoff 
between complexity and delay, whereby tuning a reduction in complexity by adjusting the frame length comes at the expense 
of delay. 

The celebrated Max- Weight policy was first introduced in |l39]| for multi-hop networks and, despite its exponential complexity 
in number of links, it provided a useful structure for designing queue length based scheduling algorithms. Later, this structure 
suggested by the Max- Weight policy lead to suboptimal but low-complexity algorithms, as well as distributed implementations 
of the Max- Weight policy for certain systems (see e.g., Greedy-Maximal network scheduling in P31 ). Our aim in proposing the 
FBDC policy and the state-action frequency framework for network scheduling is to give a structure for throughput-optimal 
algorithms for systems with time-varying channels and switchover delays, and hopefully to provide insight into designing 
scalable algorithms that can stabilize such systems. The Myopic control policies we discuss in the next section constitute a 
first approach towards characterizing the structure of some more scalable algorithms. 

E. Myopic Control Policies 

In this section, we generalize Myopic policies that we introduced for the two-queue system in Section |II] to the general 
system. Myopic policies make scheduling decisions based on queue lengths and simple channel predictions into the future. 
We present an implementation of these policies over frames of length T time slots where during the jth frame, the queue 
lengths at the beginning of the frame, Qi{jT), ...,Qj^{jT), are used for weight calculations during the frame. We describe 
the 1-Lookahead Myopic (OLM) policy for the general system in Algorithm [8] 

Algorithm 7 1-Lookahead Myopic Policy 

I: Assuming that the server is currently with queue 1 and the system is at the jth frame, calculate the following weights in 
each time slot of the current frame; 

W^iW = Qi(jr)(Ci(i)+E[Ci(i + l)|Ci(i)]) 

W,{t) ^ Q,{jT)E[C,{t + l)ia(i)] . (21) 

2: If Wi{t) > Wi{t), Vi e {2, ..., A}, then stay with queue 1. Otherwise, switch to a queue that achieves 

maxQ,(jT)E[C,(t+l)|C,(<)]. 
A similar rule applies when the server is at other queues. 

Similar to the FBDC policy, the Myopic policies can be implemented without the use of frames by setting T ~ 1, i.e., 
by utilizing the current queue lengths for updating the decision rules every time slot. This could potentially lead to more 
delay-efficient policies that are more adaptive to dynamic changes in queue lengths. We elaborate on this via the numerical 
results in the next section. 

These policies have very low complexity and they are simpler to implement as compared to the FBDC policy. As suggested 
by the simulation results in Section |IV] the Myopic policies may achieve the full stability region while providing better delay 
performance than the FBDC policy in most cases. 
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F. Myopic Control Policies 

In this section, we generalize Myopic policies that we introduced for the two-queue system in Section |II] to the general 
system. Myopic policies make scheduling decisions based on queue lengths and simple channel predictions into the future. 
We present an implementation of these policies over frames of length T time slots where during the jth frame, the queue 
lengths at the beginning of the frame, Qi{jT), ...,Qn{JT), are used for weight calculations during the frame. We describe 
the 1-Lookahead Myopic (OLM) policy for the general system in Algorithm [8] 

Algorithm 8 1-Lookahead Myopic Policy 

1: Assuming that the server is currently with queue 1 and the system is at the jth frame, calculate the following weights in 
each time slot of the current frame; 

Wi{t) = Qi{jT)(Ciit)+E[Ci{t + l)\Ci{t)]^ 

W,it) = Q,{jT)E[a{t + l)|C,(i)] . (22) 

2: If Wi{t) > Wi{t), Vz e {2, ...,N}, then stay with queue 1. Otherwise, switch to a queue that achieves 

maxQ,(jT)E[Q(t+l)|C,(i)]. 
A similar rule applies when the server is at other queues. 

The technique used for the case of two queues for analyzing the stability region achieved by the OLM policy is extremely 
cumbersome to generalize to the general system with N queues. Therefore, for the general system, we have investigated the 
performance of the OLM policy in simulations. The simulation results in Section |IV] suggest that the OLM policy may achieve 
the full stability region while providing a better delay performance as compared to the FBDC policy. 

Similar to the FBDC policy, the Myopic policies can be implemented without the use of frames by setting T = 1, i.e., 
by utilizing the current queue lengths for updating the decision rules every time slot. This could potentially lead to more 
delay-efficient policies that are more adaptive to dynamic changes in queue lengths. We elaborate on this via the numerical 
results in the next section. 

Similar to the system with two queues, the fc-Lookahead Myopic Policy is the same as before except that the following 
weight functions are used for scheduling decisions: Assuming the server is with queue 1 at time slot t, 
M/i(t) = Oi(jT)(Ci(t) + E'=iE[Ci(< + r)|Ci(f)]) and M^,(0 = ftO^) E'=i E[a(f + r)|Q(t)], z G {2, ...,iV}. 
These policies have very low complexity and they are simpler to implement as compared to the FBDC policy. 

IV. Numerical Results 

We performed simulation experiments that present average queue occupancy results for the FBDC, the One-Lookahead 
Myopic (OLM) and the Max-Weight (MW) policies for systems with iV = 2 or A^ = 3 queues. We first verified that in the 
simulation results for the FBDC policy, queue sizes grow unbounded only for arrival rates outside the stability region, and 
then performed experiments for the 1-Lookahead Myopic (OLM) policy. In all the reported results, we have A G A with 0.01 
increments. For each point at the boundary of A, we simulated one point outside the stability region. Furthermore, for each 
data point, the arrival processes were i.i.d., the channel processes were Markovian as in Fig. |2]and the simulation length was 
Ts = 100, 000 slots. 

Fig. I?] (a) presents the total average queue size, Qavg = X]tii(Qi(0 + Q2{t))/Ts, under the FBDC policy for N = 2 
queues, e = 0.25 < ec, and a frame size of T = 25 slots. The boundary of the stability region is shown by (red) lines on the 
two dimensional Ai — A2 plane. We observe that the average queue sizes are small for all (Ai, A2) G Ag and the big jumps 
in queue sizes occur for points outside A. Fig. [T] (b) presents the performance of the OLM policy with T = 25 slot frames 
for the same system. The simulation results suggest that there is no appreciable difference between the stabiUty regions of the 
FBDC and the OLM policies. Note that the total average queue size is proportional to long-run packet-average delay in the 
system through Little's law. For these two figures, the average delay under the OLM policy is less than that under the FBDC 
policy for 81% of all arrival rates considered. 

Next, we implemented the FBDC and the OLM policies without the use of any frames (i.e., for T = 1). When there are 
no frames, the FBDC policy solves the LP in Algorithm |2] in each time slot, and the OLM policy utilizes the queue length 
information in the current time slot for the weight calculations in (fT4l i. Fig. [8] (a) and (b) present the total average queue 
size under the FBDC and the OLM policies for N ~ 2 queues, T = I, and e = 0.40 > Ec- Similar to the frame based 
implementations, we observe that the average queue sizes are small for all (Ai, A2) G A for both policies and the big jumps 
in queue sizes occur for points outside A, which suggests that the the non-frame based implementation of the FBDC and the 
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Fig. 7: The total average queue size for (a) tlie FBDC policy and (b) the Myopic policy for T = 25 and e = 0.25. 

OLM policies may achieve the full stability region. The reason why the FBDC and the OLM policies provide stability without 
the use of frames is because for large queue lengths, the comer point that these policies choose to apply depend completely 
on the queue length ratios, and hence, the choice of corner points and the associated saturated-system policies utilized in the 
FBDC and the OLM policies do not change fast when the queue lengths get large. Furthermore, the no-frame implementations 
of these policies are more adaptive to dynamic changes in the queue sizes as compared to implementations with large frames. 

For the same system (i.e., N = 2 queues and e = 0.40 > ec), Fig.|9]presents the long-run packet average delay as a function 
of the sum-throughput Ai + A2 along the main diagonal line (i.e., Ai = A2). We compare the delay performance of the FBDC 
and the OLM policies with T = 1, and the Max- Weight policy which, in each time slot t, chooses the queue that achieves 
max, (5i(t)Ci(t). The maximum sum-throughput is 0.75 — e/2 = 0.55 as suggested by Theorem [3] Fig. |9] shows that while 
FBDC and the OLM policies stabilize the system for all Ai + A2 < 0.55, the system becomes unstable under the Max- Weight 
policy around Ai + A2 = 0.45. This result also confirms that the OLM policy has a much better delay performance than the 
FBDC and the Max- Weight policies. 

For N ~ 3 queues and e = 0.30, Fig.[TO]presents the long-run packet-average delay as a function of the sum-throughput J^i ^i 
along the main diagonal line (i.e., Ai = A2 = A3). The maximum sum-throughput is 1 — 0.5^ — e* (1 — 0.5^) + e*0.5'^ = 0.65 
1 — e — 0.5^^(1 — 2e) = 0.65 as suggested by Lemma [3] Similar to the previous case. Fig. [TOl shows that FBDC and the 
OLM policies stabilize the system for all J^i ^i < 0.65, the system becomes unstable under the Max- Weight policy around 
J2i ^i = 0.48. This result also confirm that the OLM policy has a delay performance than the other two policies. 

The delay results in this section show that the OLM policy is not only simpler to implement as compared to the FBDC 
policy, but it can also be more delay efficient. 

V. Conclusions 

We investigated the dynamic server allocation problem with randomly varying connectivity and server switchover time. For 
the case of two queues, we analytically characterized the stability region of the system using state-action frequencies that 
are stationary solutions to an MDP formulation for the corresponding saturated system. We developed the throughput-optimal 
FBDC policy. We also developed simple Myopic Policies that achieve a large fraction of the stability region. We extended 
the stability region characterization in terms of state-action frequencies and the throughput-optimality of the FBDC policy to 
the general system with arbitrary number of queues. We characterized tight analytical outer bounds on the stability region 
using an upper bound on the sum-rate and showed that a simple greedy-myopic policy achieves this sum-rate bound. The 
stability region characterization in terms of the state-action frequencies of the saturated system and the throughput-optimality 
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Fig. 8: The total average queue size for (a) the FBDC policy and (b) the Myopic Policy implemented without the use of frames (i.e., for T = 1) and e = 0.40. 
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Fig. 9: Delay vs Sum-throughput for the FBDC, the OLM, and the Max-Weight policies implemented without the use of frames (i.e., for T = 1) for N = 2 
queues and e = 0.40. 
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Fig. 10: Delay vs Sum-throughput for the FBDC, the OLM. and the Max-Weight policies implemented without the use of frames (i.e., for T = 1) for N = 3 
queues and e = 0.30. 
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of the FBDC policy hold for systems with arbitrary switchover times and general Markovian channels. Furthermore, the FBDC 
policy provides a new framework for developing throughput- optimal policies for network control as this policy can be used to 
stabilize a large class of other network control problems. 

In the future, we intend to explicitly characterize the stability region of systems with multiple-slot switchover times with 
general Markov modulated channels. We intend to develop throughput-optimal Myopic policies for general system models. 
Finally, joint scheduling and routing in multihop networks with dynamically changing channels and server switchover times 
is a challenging future direction. 

Appendix A-Proof of Theorem[T] 
We prove Theorem [T] for a more general system with A'^-queues and travel time between queue-i and queue- j given by Dij 
slots. We call the term J2i=i ^i/Pi the system load and denote it by p since it is the rate with which the work is entering the 
system in the form of service slots. 

Necessity 

We prove that a necessary condition for the stability of any policy is p = J2i=i ^i/Pi < 1- 

Proof: Since queues have memoryless channels, for any received packet, as soon as the server switches to some queue i, 
the expected time to ON state is l/pt- Namely, the time to ON state is a geometric random variable with parameter p,, and 
hence, 1/pi is the "service time per packet" for queue i. In a multiuser single-server system with or without switchover delays, 
with i.i.d. arrivals whose average arrival rates are Xi,i G {1, •••, N}, and i.i.d. service times independent of arrivals with mean 
1/Pi, i £ {1, •■•, N}, a necessary condition for stability is given by the system load, p, less than 1 lfT2l . ■ 

To futher ellaborate on this, consider the polling system with zero switchover times, i.i.d. arrivals of mean A; and i.i.d. service 
times of mean l/pi- The throughput region of this system is an upperbound on the throughput region of the corresponding 
system with nonzero switchover times. This is because for the same sample path of arrival and service processes, the system with 
zero switchover time can achieve exactly the same departure process as the system with nonzero switchover times by making the 
server idle when necessary. A necessary condition for the stability of the former system is p ~ Xi/pi+Xn/pn + ...+Xi/pn < 1, 
(e.g., B3l , lfT2l ). Finally, note that this necessary stability condition can also be derived by utilizing the state-action frequency 
approach of Section Hl-CI for the system with i.i.d. connectivity processes. 

Sufficiency 

Proof: Under the Gated cyclic policy, we have a Polling system with i.i.d. arrivals with mean A,;,i G {1, ...,N}, i.i.d. 
service times independent of arrivals with mean 1/pi, i £ {1, ..., N}, and finite and constant switchover delays. It is shown in 
IIT2I that the Gated cyclic policy results in an ergodic system if p ~ 'J2i=i — < 1> the expected per-message waiting times 
in steady-state are finite, and they satisfy a pseudo-conversation law. Through Little's law ( ITSl pp. 139] or [5, pp. 1109]), 
this implies that the expected number of packets in the system in steady state is finite, which in turn implies that the system 
is stable. ■ 



Appendix B-Proof of Theorem[3] 



We enumerate the states as follows: 



(23) 



s = (l,l,l) = l, s = (1,1,0) ee;2, s = (1,0,1) = 3, s = (1,0,0) = 4, 
s = (2,l,l) = 5 s = (2,1,0)eee6 s = (2,0,1) = 7 s = (2,0,0) = 8. 

We rewrite the balance equations in (fT2l i in more details. 

a;(l; 1) + x{l; 0) = (1 - e)^ {x{l; 1) + a;(5; 0)) + e(l - e){x{2; 1) + .t(6; 0)) 

+e(l-e)(x-(3;l)+a-(7;0)) + e^x{4;l) + x{8;0)) (24) 

x(2; 1) + x{2; 0) = e(l - e) (a;(l; 1) + x{b; 0)) + (1 - ef{x{2; 1) + a;(6; 0)) 

+e^{x{3; 1) + x{7; 0)) + e(l - e)(x(4; 1) + x{8; 0)) (25) 

x(5; 1) + x(5; 0) = (1 - e)^ {x{5; 1) + x{l; 0)) + e(l - e)(x(6; 1) + x{2; 0)) 

+e(l-e)(a;(7;l)+a;(3;0)) + e2(x(8; 1) + a;(4; 0)) (26) 

x{7; 1) + x{7; 0) = e(l -~ e){x{5; 1) + x(l; 0)) + e^{x{6; 1) + x{2; 0)) 

+(1 - ef{xi7; 1) + a;(3; 0)) + e(l - e){x{8; 1) + x(4; 0)) (27) 
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The following equations hold the channel state pairs (Ci,C2). 

x{l; 1) + x{l; 0) + x(5; 1) + a;(5; 0) = 1/4 (28) 

x{2; 1) + x{2; 0) + x(6; 1) + a;(6; 0) ^ 1/4 (29) 

a;(3; 1) + x(3; 0) + x(7; 1) + x{7; 0) = 1/4 (30) 

x(4; 1) + a;(4; 0) + x{8; 1) + a;(8; 0) = 1/4. (31) 

Let ui = (x(l; 1) + x{2; 1) and ii2 = {x{5; 1) + .t(7; 1)). Summing up (EDi with (US) and (OSll with (|27li we have 

eui = ~{x{l; 0) + x{2- 0)) + e(a;(3; 1) + x{4; 1)) + e{x{7; 0) + a;(8; 0)) + (1 - e){x{5; 0) + a;(6; 0)) 
eu2 = -{x{5; 0) + x{7- 0)) + e(a:(6; 1) + 2;(8; 1)) + e{x{2; 0) + x{A; 0)) + (1 - e) (a;(l; 0) + a;(3; 0)) 



Rearranging and using (l28Tl-(l3ni we have 

ui = iy^ + e(x(3; 1) + x{A; 1) + x(7; 0) + x{8; 0)) - (2 - e)(x(l; 0) + x{2; 0)) - (1 - e)(x(5; 1) + a;(6; 1)) 

(32) 

W2 - ^-^ + e{x{2; 0) - x(4; 1) + x{6; 1) - x(8; 0)) - (2 - e)(x(5; 0) + x{7; 0)) ~ (1 - e)(x(l; 1) + x{3; 1)) 

(33) 



Using ^ in (l32j and (|24ll in (l33j we have 

wi = ^ + e{x{3; 1) + x(4; 1) + x(7; 0) + a:(8; 0)) - ^^— ^ (x(4; 0) + x{8; 1)) - ii^|fc-M:^(6; 1) 



-M5;0)- ^^^'_' x(l;0)-i^-^(x(3;0)+x(7;l))-(2-6+ii-^)x(2;0) (34) 



K2 - ^ + 6(x(2; 0) + x{6; 1)) - (e + ^^^) (x-(4; 1) + x{8; 0)) - ^^ ~ f^ ^ '^'K iS; 1 



1-^ .n.n^ l + ^-^^...n^ (1 " ^r /..o. . ^ , .... nU f. , , (1 " ^) 



^(1;0)- 7' S (5;0)-i-^(a;(2;l)+x(6;0))-(2-6+i-^L)x(7;0). (35) 

e 2 — e 2 — e V 2 — e / 



e(2-6) ^ ' 6(2-e) 

Using dSOjl and dlB in OIJ and ^ in (|35]l we have 

(l-e)(3-2e) / e(l-e)\, , , , ,, 1 / . ^ . ^^ 

"1 = ^ ^(^_^) ^ + (e + ^—7^) (^-(4; 1) + 2;(8; 0)) + j—^ {x{3; 1) + x-(7; 0)) 

"' ' W^) " (' ^ ^^^) '"''"' 1) + »=(8; 0)) + 2:^ (i(2; 0) + 1(6; 1)) 

- M>^.(3;l).^.a;0,-2±l^.,5;0,-(2-..Il^).(,O,^ (37, 

Consider the LP objective function ai(^x{l; 1) + x{2; 1)) + a2{x{5; 1) + x{7] 1)), and note that the solution to this LP is 
a stationary deterministic policy for any given ai and 012. This means that, for any state s either x{s; 1) or x{s; 0) has to be 
zero. In order to maximize ai(x(l; 1) + x{2; 1)) + a2{x{5; 1) + x{7; 1)) we need 
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x(7;0) =0 if — > 



ai - (2-e)2 + (l-e)2 



a;(3; 1) = if — > 



ai - (l-e)(3-2e)' 



a;(5;0) = if — > ^ ^ 



a;(8;0) = a;(4; 1) =0 if — > 1, 

a;(6;0)=0 if — > (1 - e)(3 - 26), 

Oil 

.(i;i)^o if^>i±l^. 

ai 1 — e 



Note that we have 



(2-e)2 + (l-e)2 > (l-e)(3-2e)>l 

(2-e)2 + (l-e)2 > i±l^>l 

1 — e 

holding for all e G [0,0.5]. Consider the following two cases: 

Case-1: e > e^ = 1 - a/2/2 

In this case we have (1 — e)(3 — 2e) < (1 + e — e^)/(l — e). This means that we have the following optimal policies depending 

on the value of a2/ai. 

l<ff <(l-e)(3-26): 

©queue 1 : (1, 1, 1) : stay, (1, 1,0) : stay, (1,0, 1) : switch, (1, 0,0) : switch. 
©queue 2 : (2, 1, 1) : stay, (2, 1, 0) : switch, (2, 0, 1) : stay, (2, 0, 0) : stay. 

Substituting zero values for the state action pairs that are not chosen into (|36] | and dSTJ i. it can be seen that this policy achieves 
the rate pair 

_ (l-e)(3-2e) _ 3 - 2e 

"'' ~ 4(2 -e) ' '^' ~ 4(2 -e)' 

^>(l-e)(3-2e): 

©queue 2 : (2, 1, 1) : stay, (2, 1, 0) : stay, (2, 0, 1) : stay, (2, 0, 0) : stay. 

In this case it is optimal to stay at queue 2 for all channel conditions. The decisions at queue 1 are to switch to queue 2. 
Namely, it is sufficient that at least one state corresponding to server being at queue 1 to take a switch decision, which is the 
case for a2/ai > ((1 — e)(3 — 2e)), since x(3; 1) = if a2/ai > 1/((1 — e)(3 — 2e)). Since the poUcy decides to always 
stay at queue 2, it achieves the rate pair 

n =0, ra = 0.5. 

Note that the case for aa/ai < 1 is symmetric and can be obtained similarly. 

Case-2: e < e^ == 1 - %/2/2 

In this case we have (1 — e)(3 — 2e) > (1 + e — e^)/{l — e). This means that before the state x{Q\ 0) becomes zero, namely 
for (1 + e — e^)/(l — e) < aa/ai < (1 — e)(3 — 2e), having x{\\ 1) = is optimal. This means that there is one more corner 
point of the rate region for e < Cc. We have the following optimal policies. 



— Qi — 1 — e 



©queue 1 : (1,1,1) : stay, (1, 1,0) : stay, (1,0, 1) : switch, (1, 0,0) : switch, 
©queue 2 : (2,1,1) : stay, (2, 1,0) : switch, (2,0, 1) : stay, (2, 0,0) : stay. 
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This policy is the same policy as in the previous case and it achieves the rate pair 

(l-e)(3-2e) 3 - 2e 

^1 = 777; ^ ' ''2 - 



4(2-e) ' 4(2-6)- 

0:2 ^ l+e — e^ ■ 
ai ^ l-£ • 

We have the following deterministic actions. 

©queue 1 : (1, 1, 1) : switch, (1,1,0):?, (1,0, 1) : switch, (1, 0,0) : switch, 
©queue 2 : (2,1,1) : stay, (2,1,0):?, (2,0, 1) : stay, (2, 0,0) : stay. 

In order to find the final threshold on a2/ai, we substitute the above deterministic decisions in (IZSl l. (|26] | and dZTl i. Utilizing 
also (|28ll, (im, dlOll and (EB we obtain 

x(2; 1) = ^i^ - (1 - e)M6; 1) (38) 

:r(5;l)+x(7;l) = ^^ + ex(6;l) (39) 

The previous threshold on a-ijOLx for a;(6; 0) to be zero, i.e., (1 — e)(3 — 2e), is valid for the case where x(\\ 0) = 0. Other 
decisions staying the same, when a;(l;0) is positive and x(\\ 1) = 0, r-i increases and r^ decreases. Therefore the threshold 
on OL2loi\ for x(6;0) to be zero changes, in particular, a simple derivation shows that it becomes aijoti > (1 — e)^/e. This 
gives the following two regions: 

l+e — e~ <' £^ < (1 — e) . 

l~e — Qi — € 

The optimal policy is 

©queue 1 : (1, 1, 1) : switch, (1, 1, 0) : stay, (1, 0, 1) : switch, (1, 0, 0) : switch. 
©queue 2 : (2,1,1) : stay, (2, 1,0) : switch, (2,0, 1) : stay, (2, 0,0) : stay. 

From ( [38] l and ( [39] l it is easy to see that this policy achieves 

(1-^)^ 2-e 

''1 = -, ' ^2 = —;—■ 



02 > (i-<^)\ 

The optimal decisions at queue 1 are to switch to queue 2. This policy achieves 

©queue 2 : (2,1,1) : stay, (2, 1, 0) : stay, (2,0, 1) : stay, (2,0, 0) : stay. 

This policy achives 

n =0, ra = 0.5. 

Similar to Case-1, the case aa/^i < 1 is symmetric and can be solved similarly. 

Thus we have characterized the comer point of the stability region for the two regions of e. Using these corner points, it is 
easy to derive the expressions for the lines connecting these corner points, which are given in Theorem |3] 

Appendix C - Generalization to Non-symmetric Gilbert-Elliot Channels 

In the following, we state results analogous to the results established in Section [II] for symmetric Gilbert-Elliot channels to 
the case of non-symmetric Gilbert-Elliot channel model as given in Fig. |2] 
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Theorem 9: When the connectivity processes follow the non-symmetric Gilbert-Elliot channel model, the rate region As is 

I ^ ■,2 

the set of all rates ri > 0, r2 > that for poi < 2-^^° satisfy 

Poiri+hir2 < hi- 



Pai +P10 



(1 - PlOJri + fi.2?'2 < 1 



(Pio+Poi)^ Poi+Pio 



ri + r2 < 1 - 



(Pio+Poi)^ Poi+Pio 



n-2?-i + (l-pio)r2 < 1 



(pio+poi)^ Poi+Pio 



w/zere ft-i = (1 -poi)(l -Pio), hi = 1 +pio -Pig, and for pm > ''^2-v-,'l satisfy 



Poi +P10' 

2-pio 



Poi?'! + h^ri < hs 



Pio+Pai 
^ 1 pIo PioPni 



hsn+poiri < hs- 



(Pio+Poi)^ Poi+Pio 
Poi 



(40) 



Pio+Poi 

where h^ = (1 - pw){pw + (pio +Poi)(l -Pio))- 
Proof: We enumerate the states as follows: 

s = (l,l,l) = l, s = (l,l,0) = 2, s = (l,0,l) = 3, s = (l,0,0) = 4, 
s = (2,1,1)ee5 s = (2,1,0) = 6 s = (2,0,1) = 7 s = (2,0,0) = 8. 

We rewrite the balance equations in (fT2] i in more details. 

x(l; 1) + x{l- 0) = (1 - Piof{x{l- 1) + x(5; 0)) + poi(l - Pio)(x(2; 1) + x(6; 0)) 

+m(l-Pio)(a;(3;l)+a:(7;0)) + p2^(a;(4; 1) + x(8;0)) (41) 

a;(2; 1) + x(2; 0) = pw{l - pw){x{l- 1) + cc(5; 0)) + (1 - pio)(l - Poi)(.t(2; 1) + a;(6; 0)) 

+mPio(x(3;l)+x(7;0)) + poi(l -m)H4; 1) + x(8; 0)) (42) 

a;(5; 1) + x(5; 0) = (1 - pm)' (x(5; 1) + a;(l; 0)) + poi(l - Pio) (a;(6; 1) + x{2- 0)) 

+m(l-Pio)(a;(7;l)+a:(3;0)) + p2^(a;(8; 1) + x(4;0)) (43) 

x{l- 1) + x(7; 0) = pio(l - Pio)(a;(5; 1) + x{l- 0)) + pmPoi (x(6; 1) + x(2; 0)) 

+ (l-poi)(l-Pio)(a:(7;l)+a:(3;0)) + poi(l -Poi)(-^(8; 1) + x(4; 0)) (44) 



The following equations hold the channel state pairs (Ci,C2). 

x(l; 1) + x(l; 0) + x(5; 1) + x(5; 0) = ^°^ .„ (45) 

(Poi +Pio)^ 

x(2; 1) + x{2- 0) + x(6; 1) + x(6; 0) = , ^°'^" ,, (46) 

(Poi +Pio)^ 

x{i; 1) + x(3; 0) + x{7- 1) + x(7; 0) = . ^"^^^° ., (47) 

(Poi +Pio)^ 

a:(4; 1) + x(4; 0) + x(8; 1) + .t(8; 0) = ^i^^. (48) 
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Let Ml ^ {x{l; 1) + x{2; 1) and U2 = {x{5; 1) + x{7; 1)). Summing up (Ell with (|42]i and (|43]i with (gill we have 

pio^i = - {x{l- 0) + x{2; 0)) + poi {x{3; 1) + .t(4; 1)) + poi (2:(7; 0) + x(8; 0)) + (1 - pio) (a;(5; 0) + x(6; 0)) 
P10U2 = - (x(5; 0) + x{7; 0)) + poi {x{6; 1) + a;(8; 1)) + poi (2^(2; 0) + x{A; 0)) + (1 - pio) {x{l; 0) + a;(3; 0)) 

Rearranging and using (|45]l-(|48]l we have 

"1= ^^i^i^^+Poi {x{3; 1) + x(4; 1) + x(7; 0) + x{&; 0)) - (2 - pio)(a;(l; 0) + x(2; 0)) - (1 - pw) {x{5; 1) + a;(6; 1)) 
Poi+Pw 

(49) 

^^^ m(m+Pio^Mo) ^p^^^^^2;0)-x(4; l)+x(6; l)-a;(^ 

(50) 

Using (|43]l in (gU and ^^ in (|50ll we have 



Poi+Pw Pio(2-_pio) Pio(2-pio) 

a; 5;0 ^a:: 1;0 -(x(3; + x 7; 1 )- 2-pioH t^ x(2;0) (51) 

Pin 2 — Win Pin 2 — Pin V Pin 2 — Pin / 



Pio(2-pio) ' Pio(2-pio) ' Pio(2-pio) ' ' ^ V Pio(2-pio) 



-1 

-5 +pio(a;(2;0)+x(6;l))-(poi + z^ 1 (a;(4; l) + a;(8; 0))-(l-pio)(l + -, 

[Poi+Pior ^ Pio(2-pio)^ Pio(2-pio) 

1~P10 n n\ 1+Pl0-P?n ^r- ^^ -POl(l - Pio)^ / ,„ , ^ , ^r-nW fn , POl (1 - Pio)^ ~ 



"2= ^ ■ 7^ \-piq{x{2; 0)+x(6; 1))- poiH 77; r (a;(4; l) + a;(8; 0))-(l-pio)(l + -)a;(3; 1) 

iPm+Pio)-' V Pio(2-pio)^ Pio(2-pio) 

/o N-2;(l;0) 7T ^x{5;0) -{x{2;l) + x{6;0))- [2-P10 + 77. -]x{7;0). (52) 

pio(2-pio) Pio(2-_pio) pio(2-pio) V pio(2-pio)/ 



M2 = 



Using (Hill and (gSll in (EB and (|46ll in (l52]i we have 

Poi(l-Pio) Poi(l-Pio) P§i(l-Pio) / /. IN , ^o n^^ _l Poi /- ^o n _l ^7 ^^^ 

«i = ^ TT^ TT ^ ^+ 7:^ r(a;(4;l) + a;(8;0)) + -(x(3; 1) + 2:(7; 0)) 

Poi+Pio (2-pio)(Poi+Pio)^ Pio(2-pio) Pio(2-pio)' 

n wi , Poi(1-Pio)n /« in , 1-PlO ,^ „N l+PlO-P?o n n^ /'o , Poi(l-Plo)^\ /„ r,^ ^^:^^ 

-(l-Pio)(l + 7:5 -)a;(6;l) + rx(5;0) ^a;(l;0)- 2-pio + :r]x{2;0) (53) 

Pio(2-pio) pio(2-pio) pio(2-pio) V Pio(2-pio) / 

Poi(poi+Pio-poiPio) Poi(l-Pio)^ / , Poi(l-Pio) ^^ f. ... ,„ ^^^ Poi f ,0 n^ ^ ^fi 1^^ 

1 ; v5 7^ T7 ; n2"Ipoi + z^ r)(a^(4;l)+a;(8;0)) + -(x(2;0) + x(6; 1)) 

(Poi+Pio)" (2 -pio)(poi+pioj V pio(2-pio)>' Pio(2-pio) 

- (l-pio)(l + 7:5 r)a;(3;l) + ra;(l;0) ^a;(5; 0)- I 2-pio + -]x{7;0). (54) 

Pio(2-pio) pio(2-pio) Pio(2-pio) ^ Pio(2-pio)^ 

Consider the LP objective function ai[x{l; 1) + x{2; 1)) + a2{x{5; 1) + x{7; 1)), and note that the solution to this LP is a 
stationary deterministic policy for any given ai and a2- This means that, for any state s either x{s; 1) or x{s;0) has to be 
zero. In order to maximize ai(x(l; 1) + x(2; 1)) + a2{x{5; 1) + x{7; 1)) we need 

x(7;0) =0 if — > 



Qfi Pio(2-pio)^ +Poi(l -Pio)^' 



a;(3; 1) = if — > 



ai (1 -pio)(pio(2-pio) +Poi(l -Pio)) 
a;(5;0)=0 if — > -— 5-' 

"1 1 + PlO - PlO 

a;(8;0) =x(4;l) =0 if — > 1, 

a;(6-0)=0 if "2 ^ (1 - Pio)(pio(2 -pio) +poi(l - pio)) 
ai ~ poi 

^(1;1)=0 if ^>1+P10-P10. 

ai 1 - PlO 
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Note that we have that 

Pio(2-pio)^ +Poi(l -pwY ^ 1+Pio-Pio > ^ 

Pai ~ 1 - Pio ~ 

Pio(2-pio)^ +Poi(l -Pio)^ ^ (l-Pio)(Pio(2-pio) +Poi(l-Pio)) ^ j^ 

Poi ~ Poi ~ ' 

whenever pio + pw < 1 (the condition for positive correlation). Consider the following two cases; 



Case-l:poi> 2-pi„ 

In this case we have ^ -pio)(piq( -pio)+poi( -pioj) ^ — pio Pu) ^ 'pjjjg means that we have the following optimal policies 



depending on the value of 0,2/0,1. 



1 < H2. < (l~Pio)(Pio(2-Pio)+Poi(l-Pio)) . 
— "1 — Poi 



©queue 1 : (1, 1,1) : Stay, (1, 1,0) : stay, (1,0, 1) : switch, (1, 0,0) : switch, 
©queue 2 : (2,1,1) : stay, (2, 1,0) : switch, (2,0, 1) : stay, (2, 0,0) : stay. 



Substituting the above zero variables into (153) and (154b . it can be seen that this policy achieves the rate pair 

l-pioPoiPio+Poi(l -Pio)(Poi +P10) 1 PoiPio+Poi(l-Pio)(poi +P10) 

2-pio [pm+PwY ' 2-pio [pm+PwY 

02 -^ (l-Pio)(Pio(2-pio)+Poi(l-pio)) . 
ai poi 

©queue 2 : (2, 1, 1) : stay, (2, 1, 0) : stay, (2, 0, 1) : stay, (2, 0, 0) : stay. 

In this case it is optimal to stay at queue 2 for all channel conditions. The decisions at queue 1 are to switch to queue 2. 
Namely, it is sufficient that at least one state corresponding to server being at queue 1 to take a switch decision, which is the 

case for 02/0^ > i^-Pio)iPiai2-Pia)+Pai(l-Pio)) ^ gjjj^g ^(3. 1) = if 02/0, > ^ (^-P^oKpia(2-Pio)+Poi{l-Pio)) yl_ ^ince the 

policy decides to always stay at queue 2, it achieves the rate pair 

Ti =0, r2 = ; . 

Poi +P10 
Note that the case for a2/ai < 1 is symmetric and can be obtained similarly. 

Case-2: poi < %^ 

In this case we have (i-pi°)(pi°(^-pi°)+ppi(i-pi°)) > i+pi°-Pio ^ xhis means that before the state x(6: 0) becomes zero, namely 

POI 1-pio \ ' / .' 

for i+P^»-Pin < 0,3/0,^ < (i-Pio)(Pio(2-pio)+Poi(i-Pio)) ^ jj^yijjg ^(^. 1) = is Optimal. This means that there is one more 



l-pio ' Poi 

2-pio 



corner point to the rate region for poi < o-^" • ^^ ^^^& the following optimal policies. 



1 < H2. < l+Pio~Pio ■ 
— ai — 1— pio 

©queue 1 : (1, 1,1) : Stay, (1, 1,0) : stay, (1,0, 1) : switch, (1, 0,0) : switch, 
©queue 2 : (2,1,1) : stay, (2, 1,0) : switch, (2,0, 1) : stay, (2, 0,0) : stay. 

This policy is the same policy as in the previous case and it achieves the rate pair 

1 -PioPoiPio+Poi(l ~Pio)(poi +P10) 1 PoiPio+Poi(l -Pio)(poi +P10) 

2- Pio (Poi+Pio)^ ' 2-pio (Poi+Pio)^ 

02 -^ l+Pio-Pio . 
ai 1-pio 
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We have the following deterministic actions. 

©queue 1 : (1, 1, 1) : switch, (1,1,0):?, (1,0, 1) : switch, (1, 0,0) : switch, 
©queue 2 : (2,1,1) : stay, (2,1,0):?, (2,0, 1) : stay, (2, 0,0) : stay. 

In order to find the final threshold on a^jai, we substitute the above deterministic decisions in (|42] |. (l43T l and ( |44] |. Utilizing 
also (|45]l, (|46]l, (|42ll and (|48]l we obtain 



(1-e 



>2 



x(2; 1) = ^^^ - (1 - e)2.T(6; 1) (55) 

a;(5;l) +a:(7;l) = ; hpoia;(6;l) (56) 

The previous threshold on a2/ai for a;(6;0) to be zero, i.e., ^ -via)K'P\n[. -pio)+pou -pio)) ^ j^ y^jj^j fQj- j-jje case where 
x(l;0) = 0. Other decisions staying the same, when a;(l;0) is positive and .t(1; 1) = 0, r2 increases and r\ decreases. 
Therefore the threshold on 02/01 for a;(6; 0) to be zero changes, in particular, a simple derivation shows that it becomes 
ail Oil > (1 — Pio)(l — Poi)/poi- This gives the following two regions; 

i^^?^<^<(l-pio)(l-Poi)/poi: 

The optimal policy is 

©queue 1 : (1, 1, 1) : switch, (1, 1, 0) : stay, (1, 0, 1) : switch, (1, 0, 0) : switch. 
©queue 2 : (2,1,1) : stay, (2, 1,0) : switch, (2,0, 1) : stay, (2, 0,0) : stay. 

From (|55] | and ( |56] | it is easy to see that this policy achieves 

/, w, X PoiPio Poi(pio -PioPoi +P01) 

ri = (1 -pio)(l -poi)7 ; r^, ri - 



(Poi+Pio)^' (Poi+Pio)' 



^>(l-pio)(l-poi)/poi: 



The optimal policy is 

©queue 2 : (2,1,1) : stay, (2, 1, 0) : stay, (2,0, 1) : stay, (2,0, 0) : stay. 
The optimal decisions at queue 1 are to switch to queue 2. This policy achieves 

r\ = 0, r2 



Poi +P10 
Similar to Case-1, the case ai/ai < 1 is symmetric and can be solved similarly. 

Thus we have characterized the comer point of the stability region for the two regions of poi and pio- Using these corner 
points, it is easy to derive the expressions for the lines connecting these comer points, which are given in Theorem |9] ■ 

Closely examining the upper bound on sum-rate ri + ri, the term 1 — Pio/(pio +poi)^ is the steady state probability that 
at least one channel is in ON state. This is the maximum achievable sum-rate value for the system with zero switchover 
delay studied in ||40| . Therefore, the term p^"P°^ is exactly the loss due to switchover delay. It can be shown that, under a 
sum-rate-optimal policy, this term is equal to the steady state probability that server is at a queue with an OFF channel state 
when the other queue is at an ON channel state. 

The FBDC policy is asymptotically throughput-optimal under the non-symmetric Gilbert-Elliot channel model. This is 
straightforward as the FBDC policy only needs to solve the LP in Algorithm |2] for a given Markovian state transition stracture, 
and the non-symmetric Gilbert-Elliot channel model leads to a Markovian state transition stracture. For the non-symmetric 
Gilbert-Elliot channels case, the mappings from the queue sizes to the comer points of the rate region used by the FBDC 
policy, analogues to the mappings in tables Ull and H] can be obtained from the slopes of the lines forming the boundary of the 
stability region. Furthermore, an analysis very similar to the one in Section III-EI gives the corresponding mapping for the One 
Lookahead Myopic (OLM) policy. These mappings are shown in Table W\ for the case of poi < 2^^" ' '^^^ ^^ Table [VT] for 
the case of nni > ■ ~^"^ . 

^"^ — 2-pio 



29 



Ti(6) T*(e) T2(e) T|(e) 



T'JW T.,(e) n(,) Ti(e) If 
1^^ 1^^ ► 



corner fas \ 


corner hi \ 


corner b;3 


corner &2 


/ corner &i 


/corner bo 


(1,1.1): stay 


(1,1,1): stay 


(1,1.1): stay 


(1,1,1): .stay 


(1,1,1): switch 


(1,1,1): switch 


(1,1,0): stay 


(1,1,0): stay 


(1,1,0): stay 


(1,1,0): stay 


(1,1,0): stay 


(1,1,0): switch 


(1,0,1): stay 


(1,0,1): switch 


(1,0,1): switch 


(1,0,1): switch 


(1,0,1): switch 


(1,0,1): switch 


(1,0,0): stay 


(1,0,0): stay 


(1,0,0): stay 


(1,0,0): switch 


(1,0,0): switch 


(1,0,0): switch 


(2,1,1): switch 


(2,1,1): switch 


(2,1,1): stay 


(2,1,1): stay 


(2,1,1): stay 


(2,1,1): stay 


(2,1,0): switch 


(2,1,0): switch 


(2,1,0): switch 


(2,1,0): switch 


(2,1,0): switch 


(2,1,0): stay 


(2,0,1): switch 


(2,0,1): stay 


(2,0,1): stay 


(2,0,1): stay 


(2,0,1): stay 


(2,0,1): stay 


(2,0,0): switch 


(2,0,0): switch 


(2,0,0): switch 


(2,0,0): stay 


(2,0,0): stay 


(2,0,0): stay 



TABLE V: Mapping from the queue sizes to tlie comers of As, fco, fei, fe, ^3, ^4, fe, for poi 
optimal action is specified. The thresholds on Q2/Q1 for the FBDC policy are 0, Tj* = poi/((l 
I/T2*, T4* = l/Tj" and for the OLM policy are 0, Ti = poi/(l - pio), T2 = (1 - pio)/(2 - 
chosen in the FBDC policy if 1 < Q2/Q1 < T3*, whereas in the OLM policy if 1 < Q2/Q1 < T3. 



(i-pior 



2-pio 

-Poi)(l-pio)),T2* 

P0l),l,T3 =1/T2 



For each state s = {m{t),Ci{t),C2{t)) the 



1,T* 



= (l-pio)/(l+Pio- 

T4 = 1/Ti . For example comer 61 is 



-P?o). 



Ti 



T2 



Q2 

Qi 



corner ^3 \ 


corner ^2 


corner bi 


/ 
' corner 60 


(1,1,1): stay 


(1,1,1): stay 


(1,1,1): stay 


(1,1,1): switch 


(1,1,0): stay 


(1,1,0): stay 


(1,1,0): stay 


(1,1,0): switch 


(1,0,1): stay 


(1,0,1): switch 


(1,0,1): switch 


(1,0,1): switch 


(1,0,0): stay 


(1,0,0): stay 


(1,0,0): switch 


(1,0,0): switch 


(2,1,1): switch 


(2,1,1): stay 


(2,1,1): stay 


(2,1,1): stay 


(2,1,0): switch 


(2,1,0): switch 


(2,1,0): switch 


(2,1,0): stay 


(2,0,1): switch 


(2,0,1): stay 


(2,0,1): stay 


(2,0,1): .stay 


(2,0,0): switch 


(2,0,0): switch 


(2,0,0): stay 


(2,0,0): stay 



> 



(1-pio)' 



TABLE VI: Mapping from the queue sizes to the comers of As, boi^i: ^2, t'3, for Poi c- 9— 
optimal action is specified. The thresholds on Q2/Q1 for the FBDC policy are 0,Tj* = poi/{(l - 
(l-Pi()){PiO + (piO+Poi)(l-Pio))/poi and for the OLM policy are 0, Ti = poi/(l -Pio), 1, ^2 = 
in the FBDC policy if 1 < Q2/Q1 < T^, whereas in the OLM pohcy if 1 < Q2/Q1 < T2. 



. For each state s = (m(t), Ci(t), C2(i)) the 
Pio){piO + (Pio +poi)(l - pio))),l,T2* = 
(1 — Pio)/poi. For example comer bi is chosen 



Appendix D - Proof of Theorem[4] 

Let tk be the first slot of the A;th frame where tk+i = tfc + T. Let Di{t) be the service opportunity given to queue i at time 
slot t, where Di{t) is equal to 1 if queue i is scheduled at time slot t (regardless of whether queue i is empty or not) and 
zero otherwise. We have the following queue evolution relation: 



Q,{t + 1) = max(g.,(i) - D,{t), 0) + A,{t). 
Similarly, the following T-step queue evolution relation holds: 



(57) 



Q,{tk+T)<miixlQ,{tk] 



T-l 

-E 

T = 



A (4 




T-l 



J2Mtk + r) 



(58) 



where J2r=o ^ii^k + t) is the total service opportunity given to queue i during the fc* frame. To see this, note that if 
J2t=o Di{tk + t), the total service opportunity given to queue i during the fc* frame, is smaller than Qi{tk), then we have 
an equality. Otherwise, the first term is and we have an inequality. This is because some of the arrivals during the frame 
might depart before the end of the frame. Note that X]t=o ^ii^k + t) denotes the link i departures that would happen in 
the corresponding saturated system if we were to apply the same switching decisions over T time slots in the corresponding 
saturated system. We first prove stability at the frame boundaries. Define the quadratic Lyapunov function 

2 

i(Q(t)) = ^g,2(t), 

which represents a quadratic measure of the total load in the system at time slot t. Define the T-step conditional drift 

AT(ifc) = E [L(Q(tfc + T)) - L(Q(ifc))|Q(tfe)] , 
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where the conditional expectation is over the randomness in arrivals and possibly the scheduling decisions. Squaring both sides 
of (I58]l, using max(0, xf < a;2,Vx e N U {0}, and A(0 < 1; Vi we have 



T-l 



T = 

T-1 T-1 

2Q,:(ifc) (X)^,:(tfe + r) -^/^^tfe + r)) . (59) 



T = 



Summing dllll over the queues, using E[A,(t)2] < A^-^^^ and E[A,(ti)A,(i2)] < viEp^(^L)pE[A^(i^ < A^-^^^ for all time 
slots f 1 and i2, we can easily derive the following T-step conditional Lyapunov drift 

AT(ifc) < 2BT2 + 2TY,Q^{ik)\i 



-2^Q»(tfc)E 



T-1 



^A(tfc+T)|Q(tfc: 



.T = 



(60) 



where B = 1 + ^^^ax ^'i'^ ^^ '^^^'1 '^^e fact that the arrival processes are i.i.d. over time, independent of the queue lengths. 
Recall the definition of the reward functions fi(st,af) in in ^ and @ and let ri(st,at) be the reward function associated 
with applying policy tt* given in the definition of the FBDC policy in Algorithm |2] to the saturated system. Let ri{t) denote 
ri(si, at) for notational simplicity, i e {1, 2}. Note that ri{t) is equal to Di{t), since Di{t) is the service opportunity given to 
link i at time slot t. Now let r* = {r*)i be the infinite horizon average rate associated with policy tt*. Let x* be the optimal 
vector of state-action frequencies corresponding to tt*. Define the time-average empirical reward from queue i in the saturated 
system, fT.i{tk), i G {1,2} by 



1 _ 



T = 

Similarly, define the time average empirical state-action frequency vector XT(ife;s,a) 

tk+T~l 



XT(ife;S,a) = — ^ /{s,=s,a,=a}, (61) 

where Ie is the indicator function of an event E, i.e., /^ = 1 if £■ occurs and J^; = otherwise. Using the definition of the 
reward functions in (|3} and dUi, we have that 

rrAtk) = ^^ri(s,a)xT(ifc;s,a), i € {1,2}, 

seS a 

and Yriik) — {fT,i{tk))i- Similarly, we have 

sG5 a 

Now we utilize the following key MDP theory result in Lemma 4.1 1271 . which states that as T increases, xxitk) ~ 
(xT(tfc;s,a))s^a converges to x*. 
Lemma 4: For every choice of initial state distribution, there exists constants ci and C2 such that 

P(PT(ifc) - x*|| > Jo) < cie-=^*°^, VT > 1, V5o > 0. 

Furthermore, convergence of 'kritk) to x* is with probability (w.p.) 1. 

This result applies in our system because every extreme point x* of X can be attained by a stationary and deterministic policy 

that has a single irreducible recurrent class in its underlying Markov chain 1271 . 1351^1 Due to the linear mapping from the 

state-action frequencies to the rewards, by Schwartz inequality, each component of rT{tk) also converges to the corresponding 

component of r* . Therefore, we have that for every choice of initial state distribution, there exists constants ci and C2 such 

that 

IM-tk) - r*|| > S,) < cie-^^^i^, VT > l,V<5i > 0. (62) 



'Note that in general multiple stationary-deterministic policies can yield the same optimal reward vector r*. Among these, we choose the one that forms 
a Markov chain with a single recuiTent class. 
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Furthermore, convergence of iritk) to r* is w.p. 1. Now let Rritk) = J2iQt{tk)fT,i{tk) and R*{tk) = J2iQi{tk)r*. We 
rewrite the drift expression: 

^||^< BT+Y,Q^itk)X^ -E[RT{tk)\Q{tk)] 

% 

+ E[i?*(tfe)-i?T(tfc)|Q(tfc)]. (63) 

Now we bound the last term. For all (^2 > we have 

. P {R*{tk) - Rritk) > S2\\Cl{tk)\\ |Q(ifc)) 
+E[R*{tk)~RT{tk)\Q{tk),R*{tk)^RT{tk)<S2\\Cl{tk)\\] 
. P (i?*(tfc) - Rritk) < <52||Q(tk)|| |Q(tfc)) 

< {Y.Q^{tk))P{\R*{h)~RT{tk)\>S2\\Q{tk)\\ |Q(tfe)) 

+ S2\\Q{tk)\\, (64) 

where we bound the first expectation by J^i Qi{tk) by using ||r*|| < 1, the second expectation by (52||Q(ife)|| and the second 
probability by 1. By Schwartz inequality we have 

P {\R*{tk) - RT{tk)\ > '52||Q(tk)|| \Q{tk)) 

< P(||r*-fT(ifc)||>(52|Q(ifc)). (65) 

Using ( |62] | and (l65T l in ( |64] |, we have 

E[R*{tk)^R^{tk)\Q{tk)]<(^Q^{tk))cle-''''"-^+52\\Q{tk)\\■ 

i 

Hence, using ||Q(t/e)|| < ^iQi{ik)^ we bound < [63] t as 



^Q*(4-) 



cie 



-35.^T^J^^ 



Therefore, calling S = cie '^^^2'^ + $2, we have 

AT(tfe) 



2T 



<BT+J2 Q.{tk)Xr~J2 Q^itk)r*+SY, Q^itk). 



(66) 



Now for A strictly inside the 5-stripped stability region A*, there exist a small ^ > such that A + f .1 = r — 51, for some 
r e Ag. Utilizing this and the fact that ^ • Qi{t){ri — r*) < by definition of the FBDC policy in Algorithm [2] we have. 



2r 



< 



BT-(^o,(t,))e. 



(67) 



Therefore, the queue sizes have negative drift when ^^ Qi{tk) is larger than BT/^. This establishes stability of the queue sizes 
at the frame boundaries t = kT, k = {0, 1, 2, ...} for A within the 5-stripped stability region Af . To see this, take expectations 
with respect to Q,{tk) to have 



[L(Q(tfc+i))]-E[L(Q(ifc))] < 2BT^-2^TE 



X!q«(^* 



Writing a similar expression over the frame boundaries tk,k G {0, 1, 2, ..., K}, summing them and telescoping these expressions 
leads to 



K-l 



LiQitK)) - i(Q(0)) < 2KBT^ - 2^T^E 



fc=0 



EQ»(^fc) 
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Using L{Q{tK)) > and L(Q(0)) = 0, we have 

A'-l 



BT 

< —— < oo. 



1 ;^^ 

This implies that 

limsup-^ Y^ ^E[Q,(tfc)] < ^ < oo. 



1 ^^-sr-^.^ , ., BT 



This estabHshes stability (as defined in Definition [T]) at the frame boundaries tk, k E {0, 1, 2, ...}. 

For any given time t S {tk,tk+i) we have Qi{t) < Qi{tk) + I]t=o ^^(*fc + '^)- Therefore, E[(5j(t)] < ¥.[Qi{tk)] + 
TXi < E[Qi{tk)] + TAmax- Hence stability at the frame boundaries implies the overall stability of the system. Finally, 
S = cie~'^^^^'^ + 62 for any 62 > 0. Therefore choosing 62 appropriately (for example, S2 = 7^-0. s+fc ^^j. ^Qj^^g small ^3 > 0), 
we have that 6{T) is a decreasing function of T. Therefore, for any S > 0, one can find T such that the hypothesis of the 
theorem holds. 

Appendix E - Proof of Lemma[2] 
We first establish that 

Considering the mappings in tables |III] and |IV| for the regions of e where the OLM policy and the optimal policy "choose" 
the same corner point, we have ^ = 1. In the following we analyze the ratio ^ in the regions where the two policies choose 
different corner points, which we call "discrepant" regions. We will use Qi and Q2 instead of Qi{t) and Q2{t) for notational 
simplicity. We first consider the case Q2 > Qi, and divide the proof into separate cases for different regions of e values. 

Weighted Departure-Rate Ratio Analysis, Case 1: e < Ec 

Note that the following inequality always holds: j^f > (iSj) ■ However, we have j^f = ~J^ for e = e^ = 0.245 for the 
case of e < ec = 0.293. 

Case 1.1: e < et 

For this case we have j^f < ~^ ■ 

Discrepant Region 1: ^ ^'^' < q < ^— ^ 

In this case the OLM policy chooses the comer point 61 whereas the optimal policy chooses the comer point bo. Therefore, 



Q2\ - 2+ 2 1-e 



= 1 > 0.9700. 

2 - 

Discrepant Region 2: ^^^^zf^ < §7 < f^ 

In this case the OLM policy chooses the comer point 62 whereas the optimal policy chooses the comer point 61. Therefore, 



'SUS 2 + 8(2-e)) + ^2(8 8(2-e) ) 



3 _ e , £ I Q2 /' 3 



= 8 2 8(2.)'Q,V8 8(2-^)^ > 0.9002. 
{^-(-Y I Q2 n _ e\ - 

4 "^ Qi V2 4/ 

This is a minimization of a function of two variables for all possible e values in the interval < e < e^, and the ratio ^ in 
the interval ii±li:l!l < Si < 2^. 

1 — e Q\ 1 — e 

CASE 1.2: et < e < ec 

For this case we have 7^^ > ^ ~'^' 

1 — e — € 
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Discrepant Region 1: L_^^ ^ o ^ '^~e^ 

In this case the OLM policy chooses the comer point 61 whereas the optimal policy chooses the comer point 60 • Therefore, 



Q,i - 2+ 2 1- 



= 1 > 0.9500. 

2 - 

Discrepant Region 2: ^ ^^' < q < jEf 

In this case the OLM policy chooses the comer point 62 whereas the optimal policy chooses the comer point b^. Therefore, 

Q22 

> (i^) (l-e+ — ^) + - - — ^ > 0.9150. 
-4-e^M 4(2-e)^ 4 4(2 - e) " 

Discrepant Region 3: - — j^'^'^ ' < n^ < ~'^ 

In this case the OLM policy chooses the comer point 62 whereas the optimal policy chooses the comer point bi. Therefore, 



* = 



l(^^)+Q2(i-f) 



3 i. J- e _|_ Q2 ( 3 e '\ 

> 8 2 ^ 8(2^ "^ Qils W^l > Q 9474 

4 ^ Qi V2 4/ 

Due to symmetry, the same bounds on 'I' applies for Q2 < Qi- 
Weighted Departure-Rate Ratio Analysis, Case 2: e> ec 

For the case where e > ec, we have (1 — e)(3 — 2e) < (1 — e)/e and ^-^ < j^. Therefore, the only discrepant region between 

the FBDC and the OLM policies for Q2 > Qi is given by (1 - e)(3 - 2e) < S^ < ^, where for this interval the OLM 

policy chooses the corner point 61, whereas the FBDC policy chooses the corner point bo. 

Discrepant Region 1: (1 - e)(3 - 2e) < ^i < i^ 

In this case the OLM policy chooses the comer point 61 whereas the optimal policy chooses the comer point 60 • Therefore, 

Qld ~ f + W=7)) +<52(8 - 8(2^) 



Q25 

e 

4(2 -e)^ ' 4 4(2 -e) 



'■2 



Due to symmetry, the same bound on 'f applies for Q2 < Qi- Combining all the cases, for all e e [0,0.5], we have that 
* > 0.90 for all possible Qi and Q2. 

Now the following drift expression for the OLM policy can be derived similarly to the derivation of (l6Ft used in the proof 
of Theorem H) 

^^<BT+Y,Q^{tk)X^-Y,Q^itk)h+S4Y,Q^{tk), 

i i i 

where 5^{T) is a decreasing function of T. Using ( fTSl l 

AT(ifc) 



2r 



<BT+Y^Q,{tk)K-Q.^Q^{tk)r*+5^Q,{tk). 



Using an argument similar to that for (|7?t we have that for (Ai, A2) strictly inside the 0.9 fraction of the 64-stripped stability 
region, there exist a small ^ > such that (Ai, A2) + {£,,0 — 0.9(ri,r2) — (5^, 84), for some r = (ri,r2) G Ag. Substituting 
this expression for (Ai, A2) and using ^^ Qiit){r — r*) < we have, 

% 
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<BT-(j2Q,{t,))c 



After cancelations we have. 

At fa) 
2T 

Therefore, using an argument similar to the proof of Theorem |4] in Appendix C, the system is stable for arrival rates within 
at least the 0.9 fraction of 5i-stripped stability region, where 5i{T) is a decreasing function of T. 

Appendix F - Proof of Lemma[3] 

We follow similar steps to the proof of the sum-throughput upper bound for the case of two queues in Appendix A. In order 
to obtain an expression for r^, i e {1, ..., A^}, we sum the 2^^^ equations in ( [TtI i for which the server location in is i and the 
channel process of queue i, Ci, is 1. This gives for all i S {1, ...,-/V} 

Piori= - ^ ^x{s,a) + Poi^ x(s;z) 

s: m— 2 a^i s: m—i 

C. = l C,=0 

+ (1-Pio) ^ x(s;i) + poi ^ x(s;i). 

s: m^i s: m^i 

C,=l C,=0 

Summing r^ over all queues and using the normalization condition J^s J2a ■"■(*' '^) ~ ^^ ^^ '^^^^ 

N N 

i—1 i— 1 i^i s:ni—i 

Ci = l,Cj=0 

N 



{poi+pw)^Yl Yl ^i^'^j) 



z— 1 j^i s: m^i 

Ci = l,C.,=l 

N 



(i-poi-pio)5m H x(s;i)- 



i— 1 7:^2 s: m— 2 
■'^ Ci=0,C, = l 



From Corollary [T] there exists a stationary-deterministic policy tt that solves this LP of maximizing J^i ^i{^) over the state- 
action polytope X. Therefore, under this policy tt, at each state, at least one of the actions must have state-action frequency. 
Therefore, in order to maximize the sum-rate, the terms that have negative contribution to the sum-rate must be zero: 



TV TV 



{Pio + Poi)^ri :^ poi + {I - poi - pio)^^ ^ x(s;j). (68) 



i— 1 i—1 i^i s: rn^i 



Similar to the two-queue case in Appendix A, we utilize the expressions resulting from the fact that the steady state probability 
of each channel state vector is known. For instance, for Cq = P((Ci, ..., Cjv) = (0, ..., 0)) = -, — r^^-^nv > we have 

N 

E E Ex(,a)=Cr. 



Summing these expressions we obtain 



z— 1 s: m—i a^A 



N N 

EE E x(.;,) = l-Cr-^ 

i—1 i^i s: m—i i—1 

Ci=0,Cj=l 



r,. 



Combining this expression with ( |68] l we obtain 

N 

Y.r.-i-cr-{pMi-cr)-poicn- 
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Appendix G - Proof of Theorem[8] 

This proof follows very similar lines to the proof of Theorem 2] in Appendix D. The following T-step conditional Lyapunov 
drift expression can easily be derived similar to (|60). 

AT{tk) < NBT^ + 2TJ2Q^itk)X^ 

i=l 
TT-l 

^A(tfe + T)|Q(i,), 

.r=0 

where 5 = 1 + A^^ax- Recall the definition of the reward functions ri(st, a^), i G {1, •■■, N}, in (fT6] l and let rj(st, a^) be the 
reward function associated with applying policy tt* given in the definition of the FBDC policy in Algorithm |6] to the saturated 
system. Let ri{t) denote ^^(st, a^) for notational simplicity, i E {1, ..., N}. Note again that ri{t) is equal to Di{t), since Di{t) 
is the service opportunity given to link i at time slot t. Now let r* = {r*)i be the infinite horizon average rate associated with 
policy TT*. Let x* be the optimal vector of state-action frequencies corresponding to tt*. Define the time-average empirical 
reward from queue i in the saturated system, rT.i(ifc), i G {1, ..., N} by 



-2^Q.(tfe)E 



rT,z{tk) = 7f;^ri{tk + T). 



T = 

Similarly, define the time average empirical state-action frequency vector XT(ifc;s,a). 

, tfc+T-l 
XT(ifc;S,a) = — 22 -^{sx=s,a,=a}, 

T=tk 

where Ie is the indicator function of an event E, i.e., /^ = 1 if £■ occurs and Ie = otherwise. Using the definition of the 
reward functions in (fT&t . we have that 

and Tritk) ~ {fT.i{tk))i- Similarly, we have 

'^*^ = im^»(S:a)x*(s,a), ^ e {1,...,^}. 

se5aG^ 

Again utilizing Lemma 4.1 in |f27], we have that for every choice of initial state distribution, there exists constants ci and C2 
such that 

P(||fT(tfc) - r*|| > S,) < cie-^^^i^, VT > l,V<5i > 0. (69) 

Furthermore, convergence of rritk) to r* is w.p. 1. Now let Rritk) == J2iQii'tk)fTjitk) and R*{tk) = J^iQii^kV*. We 
rewrite the drift expression: 

Aritk) / NBT 



< -^+J2Q,{tk)X;-E[RT{tk)\Cl{tk 

i 

2 ^Y.Q^{tk)X.-Y.Q.{tk)r: 



2T - 2 

NBT 



+ E[R*{tk)-RTitk)\Q{tk)]- (70) 

The last term can be bounded similarly to (|64t to have for all 62 > 

E[R*{tk)-RTitk)\Q{tk)] < (^g.(tfe))P(|i?*(ifc)-^T(ife)|><52||Q(ifc)ll|Q(<fe)) +'52||Q(tfc)||. (71) 

i 

By Schwartz inequality we have 

P {\R*{tk) - RT{tk)\ > '52||Q(tk)|| \Q{tk)) 

< P(||r*-fT(ifc)||>(52|Q(ifc)). (72) 

Using ( |69] l and ( |72] i in ( TTTT i, we have 

E[R*{tk)^R^{tk)\Q{tk)]<(^Q^{tk))cle-^''^^^+52\\Q{tk)\\■ 

i 
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Hence, using ||Q(ife)|| < ^iQi{tk), we bound dTOl i as 



kK 



Therefore, calling S = cie '^^^2'^ + S2, we have 

i i i 

Now for A strictly inside the 5-stripped stability region A^, there exist a small ^ > such that A + ^.1 = r — 51, for some 
r G As. Utilizing this and the fact that ^- Qi{t){ri — r*) < by definition of the FBDC policy in Algorithmic we have, 

i 

Therefore, the queue sizes have negative drift when ^^ Qi{tk) is larger than ^^'^ . This establishes stability of the queue sizes 
at the frame boundaries t — kT, k — {0, 1, 2, ...} for A within the 5-stripped stability region A* (see e.g., 1321 Theorem 3]). 
For any given time t e {tk,tk+i) we have Qi{t) < Qi{tk) + I]t=o ^»(*fe + '^)- Therefore, 'E[Qi{t)] < E[Qi{tk)] + TXi < 
E[Qi(ifc)] +TA,nax- Hence stability at the frame boundaries implies the overall stability of the system since the frame length 
is constant. Finally, 5 — cie'^'-^^^T j^ §^ for any 52 > 0. Therefore choosing 52 appropriately (for example, 52 = T^"-^+'^-' for 
some small ^3 > 0), we have that 5{T) is a decreasing function of T. Therefore, for any 5 > 0, one can find T such that the 
hypothesis of the theorem holds. 
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